[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:53:52 dist-test-slave-rwrg krb5kdc[14214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:53:52 dist-test-slave-rwrg krb5kdc[14214](info): set up 2 sockets
Aug 06 15:53:52 dist-test-slave-rwrg krb5kdc[14214](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 15:53:53.664839 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.383s user 0.002s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:53:53 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:53:53.798342 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33381
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:33381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:53.923507 14231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:53.923786 14231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:53.923831 14231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:53.927861 14231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:53:53.927928 14231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:53.927953 14231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:53.927973 14231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:53:53.928066 14231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:53:53.933620 14231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:33381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:53.935115 14231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:53.936487 14231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:53.949211 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:53.949211 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:53.950757 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:53.951745 14231 server_base.cc:1034] running on GCE node
I20260806 15:53:53.952819 14231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:53.954509 14231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:53.955753 14231 hybrid_clock.cc:648] HybridClock initialized: now 1786031633955699 us; error 66 us; skew 500 ppm
Aug 06 15:53:53 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031633, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:53.959048 14231 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:53:53.960448 14231 webserver.cc:460] Webserver started at http://127.13.223.62:45213/ using document root <none> and password file <none>
I20260806 15:53:53.961404 14231 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:53.961525 14231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:53.961927 14231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:53.964702 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "c3e00c66106e46c39a1c296e74c938f3"
format_stamp: "Formatted at 2026-08-06 15:53:53 on dist-test-slave-rwrg"
server_key: "1e2b3a8a4dbe02dbe0a4ecca3754877f"
server_key_iv: "7c5be95631fa2ce1ebf901d36ef028d4"
server_key_version: "encryptionkey@0"
I20260806 15:53:53.965356 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "c3e00c66106e46c39a1c296e74c938f3"
format_stamp: "Formatted at 2026-08-06 15:53:53 on dist-test-slave-rwrg"
server_key: "1e2b3a8a4dbe02dbe0a4ecca3754877f"
server_key_iv: "7c5be95631fa2ce1ebf901d36ef028d4"
server_key_version: "encryptionkey@0"
I20260806 15:53:53.970494 14231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 15:53:53.974146 14261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:53.975798 14231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:53:53.976027 14231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "c3e00c66106e46c39a1c296e74c938f3"
format_stamp: "Formatted at 2026-08-06 15:53:53 on dist-test-slave-rwrg"
server_key: "1e2b3a8a4dbe02dbe0a4ecca3754877f"
server_key_iv: "7c5be95631fa2ce1ebf901d36ef028d4"
server_key_version: "encryptionkey@0"
I20260806 15:53:53.976188 14231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:54.005198 14231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:54.009328 14231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:54.009598 14231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:54.025667 14231 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:33381
I20260806 15:53:54.025729 14341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:33381 every 8 connection(s)
I20260806 15:53:54.027097 14231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:53:54.032027 14342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:53:54.036839 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14231
I20260806 15:53:54.037032 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:53:54.038451 14204 external_mini_cluster.cc:1428] Setting key 340110a0679428f1ca8ec6e01d7ead55
I20260806 15:53:54.040609 14342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3: Bootstrap starting.
I20260806 15:53:54.045298 14342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3: Neither blocks nor log segments found. Creating new log.
I20260806 15:53:54.046627 14342 log.cc:824] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3: Log is configured to *not* fsync() on all Append() calls
I20260806 15:53:54.049328 14342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3: No bootstrap required, opened a new log
I20260806 15:53:54.054044 14342 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e00c66106e46c39a1c296e74c938f3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33381 } }
I20260806 15:53:54.054309 14342 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:53:54.054421 14342 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e00c66106e46c39a1c296e74c938f3, State: Initialized, Role: FOLLOWER
I20260806 15:53:54.055136 14342 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e00c66106e46c39a1c296e74c938f3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33381 } }
I20260806 15:53:54.055344 14342 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:53:54.055486 14342 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:53:54.055661 14342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:53:54.057017 14342 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e00c66106e46c39a1c296e74c938f3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33381 } }
I20260806 15:53:54.057617 14342 leader_election.cc:302] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3e00c66106e46c39a1c296e74c938f3; no voters:
I20260806 15:53:54.057974 14342 leader_election.cc:288] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:53:54.058147 14356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:53:54.058400 14356 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [term 1 LEADER]: Becoming Leader. State: Replica: c3e00c66106e46c39a1c296e74c938f3, State: Running, Role: LEADER
I20260806 15:53:54.058877 14356 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e00c66106e46c39a1c296e74c938f3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33381 } }
I20260806 15:53:54.059763 14342 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:53:54.061965 14358 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3e00c66106e46c39a1c296e74c938f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e00c66106e46c39a1c296e74c938f3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33381 } } }
I20260806 15:53:54.062235 14358 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [sys.catalog]: This master's current role is: LEADER
I20260806 15:53:54.061932 14357 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3e00c66106e46c39a1c296e74c938f3. Latest consensus state: current_term: 1 leader_uuid: "c3e00c66106e46c39a1c296e74c938f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e00c66106e46c39a1c296e74c938f3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33381 } } }
I20260806 15:53:54.062520 14357 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3 [sys.catalog]: This master's current role is: LEADER
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:54.063359 14375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:53:54.067842 14375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:53:54.073314 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.043735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.13.223.62:33381)
0806 15:53:54.044357 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:54.044382 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:54.045992 (+ 1610us) server_negotiation.cc:407] Connection header received
0806 15:53:54.050048 (+ 4056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:54.050064 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:54.050656 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:54.051028 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:54.053744 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.055300 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.057446 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.058214 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.058464 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.064673 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:54.064713 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:54.064730 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:54.064765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:54.067531 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:54.068427 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:54.068447 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:54.068453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:54.068687 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:54.069337 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:54.069341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:54.069343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:54.070119 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:54.070355 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:54.071241 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 15:53:54.071598 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":230,"threads_started":1}
I20260806 15:53:54.075408 14375 catalog_manager.cc:1347] Generated new cluster ID: 9c8ebe2d9a2c4f1da13664602f59f2ac
I20260806 15:53:54.075480 14375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:53:54.083113 14375 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:53:54.083997 14375 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:53:54.103380 14375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3e00c66106e46c39a1c296e74c938f3: Generated new TSK 0
I20260806 15:53:54.104218 14375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:53:54.141654 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33381
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:54.306231 14389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:54.306474 14389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:54.306521 14389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:54.310686 14389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:54.310741 14389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:54.310802 14389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:53:54.316300 14389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:54.317368 14389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:54.318378 14389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:54.330264 14395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:54.330325 14396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:54.330286 14389 server_base.cc:1034] running on GCE node
W20260806 15:53:54.330490 14398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:54.331048 14389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:54.331761 14389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:54.332954 14389 hybrid_clock.cc:648] HybridClock initialized: now 1786031634332937 us; error 34 us; skew 500 ppm
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031634, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:54.336777 14389 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:53:54.337802 14389 webserver.cc:460] Webserver started at http://127.13.223.1:37189/ using document root <none> and password file <none>
I20260806 15:53:54.338410 14389 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:54.338507 14389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:54.338747 14389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:54.341109 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "9d1c381870ca41c78f4850b518a55531"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "e873d31e117d0a6408e0dae2c83f5914"
server_key_iv: "8e24c0d066e02d14610de6edf3908063"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.341778 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "9d1c381870ca41c78f4850b518a55531"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "e873d31e117d0a6408e0dae2c83f5914"
server_key_iv: "8e24c0d066e02d14610de6edf3908063"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.345820 14389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 15:53:54.348812 14408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.350047 14389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:53:54.350205 14389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "9d1c381870ca41c78f4850b518a55531"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "e873d31e117d0a6408e0dae2c83f5914"
server_key_iv: "8e24c0d066e02d14610de6edf3908063"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.350325 14389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:54.365146 14389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:54.369055 14389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:54.369333 14389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:54.370195 14389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:53:54.371297 14389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:54.371369 14389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.371439 14389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:54.371490 14389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.393780 14389 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:44641
I20260806 15:53:54.393882 14590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:44641 every 8 connection(s)
I20260806 15:53:54.394793 14389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:53:54.401193 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14389
I20260806 15:53:54.401316 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:53:54.401580 14204 external_mini_cluster.cc:1428] Setting key c259f9343b57204e22caf0c8e215733e
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031634, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:54.412439 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.397312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:50007 (local address 127.13.223.62:33381)
0806 15:53:54.397472 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:54.397495 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:54.398651 (+ 1156us) server_negotiation.cc:407] Connection header received
0806 15:53:54.400333 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:54.400336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:54.400397 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:54.400502 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:54.402104 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.402830 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.404259 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.404700 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.404825 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.407876 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:54.407883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:54.407889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:54.407922 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:54.409859 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:54.410465 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:54.410471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:54.410476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:54.410594 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:54.411001 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:54.411005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:54.411009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:54.411273 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:54.411408 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:54.411990 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 15:53:54.412225 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 15:53:54.413154 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.397689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:33381 (local address 127.13.223.1:50007)
0806 15:53:54.398452 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:54.398496 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:54.400112 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:54.400704 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:54.400710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:54.401190 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:54.401908 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:54.401925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.402972 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.402975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:54.404118 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:54.404128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.404843 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.404846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:54.405070 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.405804 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:54.405821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:54.407707 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:54.409992 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:54.410010 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:54.410024 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:54.410331 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:54.410734 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:54.410737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:54.410738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:54.410886 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:54.411401 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:54.411406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:54.411651 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 15:53:54.411817 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 15:53:54.412111 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":266,"threads_started":1}
I20260806 15:53:54.414659 14591 heartbeater.cc:347] Connected to a master server at 127.13.223.62:33381
I20260806 15:53:54.415001 14591 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:53:54.415599 14591 heartbeater.cc:511] Master 127.13.223.62:33381 requested a full tablet report, sending...
I20260806 15:53:54.417773 14285 ts_manager.cc:194] Registered new tserver with Master: 9d1c381870ca41c78f4850b518a55531 (127.13.223.1:44641)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:53:54.420456 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:50007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:53:54.435099 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33381
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:54.599220 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:54.599546 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:54.599635 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:54.604586 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:54.604668 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:54.604773 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:53:54.611021 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:54.612109 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:54.613240 14599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:54.624584 14608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:54.624634 14607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:54.624869 14611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:54.625031 14599 server_base.cc:1034] running on GCE node
I20260806 15:53:54.625484 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:54.626132 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:54.627305 14599 hybrid_clock.cc:648] HybridClock initialized: now 1786031634627263 us; error 52 us; skew 500 ppm
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031634, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:54.630892 14599 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:53:54.631875 14599 webserver.cc:460] Webserver started at http://127.13.223.2:45109/ using document root <none> and password file <none>
I20260806 15:53:54.632462 14599 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:54.632591 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:54.632825 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:54.635075 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "c2bc102f1f7447cdb4c0683a6690c5a1"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "da43574bd8438bcf47ce1de034b6b4bc"
server_key_iv: "79d9ffd4e136104c0d2a52916bbdf9d7"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.635715 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "c2bc102f1f7447cdb4c0683a6690c5a1"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "da43574bd8438bcf47ce1de034b6b4bc"
server_key_iv: "79d9ffd4e136104c0d2a52916bbdf9d7"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.639937 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:53:54.642748 14622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.644098 14599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:53:54.644376 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "c2bc102f1f7447cdb4c0683a6690c5a1"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "da43574bd8438bcf47ce1de034b6b4bc"
server_key_iv: "79d9ffd4e136104c0d2a52916bbdf9d7"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.644533 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:54.674386 14599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:54.678081 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:54.678371 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:54.679247 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:53:54.680312 14599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:54.680382 14599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.680455 14599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:54.680528 14599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 15:53:54.702078 14599 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:35871
I20260806 15:53:54.702179 14805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:35871 every 8 connection(s)
I20260806 15:53:54.703231 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:53:54.713526 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14599
I20260806 15:53:54.713642 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:53:54.713973 14204 external_mini_cluster.cc:1428] Setting key f0697d61f269a1e56de437ca1e9c9e96
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031634, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:54.719424 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.705552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:42071 (local address 127.13.223.62:33381)
0806 15:53:54.705707 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:54.705711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:54.706583 (+ 872us) server_negotiation.cc:407] Connection header received
0806 15:53:54.708263 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:54.708267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:54.708363 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:54.708466 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:54.709999 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.710551 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.711903 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.712142 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.712254 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.715539 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:54.715578 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:54.715580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:54.715603 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:54.717334 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:54.717888 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:54.717892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:54.717895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:54.717979 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:54.718318 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:54.718322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:54.718323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:54.718500 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:54.718614 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:54.719137 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 15:53:54.719262 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:53:54.720376 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.705756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:33381 (local address 127.13.223.2:42071)
0806 15:53:54.706393 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:54.706438 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:54.708017 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:54.708689 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:54.708696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:54.709181 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:54.709839 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:54.709856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.710700 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.710704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:54.711769 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:54.711779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.712308 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.712313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:54.712562 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.713223 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:54.713239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:54.715364 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:54.717466 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:54.717471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:54.717482 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:54.717780 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:54.718076 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:54.718079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:54.718081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:54.718217 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:54.718613 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:54.718619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:54.718864 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 15:53:54.719093 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 15:53:54.719409 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":185,"threads_started":1}
I20260806 15:53:54.721802 14808 heartbeater.cc:347] Connected to a master server at 127.13.223.62:33381
I20260806 15:53:54.722071 14808 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:54.722563 14808 heartbeater.cc:511] Master 127.13.223.62:33381 requested a full tablet report, sending...
I20260806 15:53:54.723711 14285 ts_manager.cc:194] Registered new tserver with Master: c2bc102f1f7447cdb4c0683a6690c5a1 (127.13.223.2:35871)
I20260806 15:53:54.724272 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:42071
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:53:54.745965 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33381
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:54.904803 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:54.905077 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:54.905128 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:54.909408 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:54.909467 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:54.909600 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:53:54.915120 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:54.916193 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:54.917269 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:54.927899 14829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:54.928156 14813 server_base.cc:1034] running on GCE node
W20260806 15:53:54.927951 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:54.927901 14827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:54.928774 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:54.929394 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:54.930593 14813 hybrid_clock.cc:648] HybridClock initialized: now 1786031634930583 us; error 27 us; skew 500 ppm
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031634, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:54.934029 14813 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:53:54.934859 14813 webserver.cc:460] Webserver started at http://127.13.223.3:40187/ using document root <none> and password file <none>
I20260806 15:53:54.935364 14813 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:54.935420 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:54.935595 14813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:54.937819 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "c19e6f1602494afc85b5ebd9cdf74a14"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "261e475f559b615eefff3032445db38d"
server_key_iv: "21ccde39fbd965ec7f8db88edf0a2afe"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.938380 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "c19e6f1602494afc85b5ebd9cdf74a14"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "261e475f559b615eefff3032445db38d"
server_key_iv: "21ccde39fbd965ec7f8db88edf0a2afe"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.942548 14813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:53:54.945185 14841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.946614 14813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:53:54.946720 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "c19e6f1602494afc85b5ebd9cdf74a14"
format_stamp: "Formatted at 2026-08-06 15:53:54 on dist-test-slave-rwrg"
server_key: "261e475f559b615eefff3032445db38d"
server_key_iv: "21ccde39fbd965ec7f8db88edf0a2afe"
server_key_version: "encryptionkey@0"
I20260806 15:53:54.946813 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:54.960335 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:54.964178 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:54.964385 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:54.965197 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:53:54.966149 14813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:54.966197 14813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.966270 14813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:54.966311 14813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:54.984812 14813 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:33957
I20260806 15:53:54.984905 15022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:33957 every 8 connection(s)
I20260806 15:53:54.985755 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:53:54.994560 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14813
I20260806 15:53:54.994736 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:53:54.995065 14204 external_mini_cluster.cc:1428] Setting key 0c346d757fb14b74c5d51a186e7799a7
Aug 06 15:53:54 dist-test-slave-rwrg krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031634, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:55.003371 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.987658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:35891 (local address 127.13.223.62:33381)
0806 15:53:54.987843 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:54.987847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:54.988868 (+ 1021us) server_negotiation.cc:407] Connection header received
0806 15:53:54.991359 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:54.991362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:54.991411 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:54.991488 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:54.993792 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.994343 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.995723 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.996117 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.996237 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.999049 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:54.999056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:54.999062 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:54.999107 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:55.001116 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:55.001699 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:55.001705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:55.001710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:55.001824 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:55.002312 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:55.002314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:55.002315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:55.002569 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:55.002661 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:55.003125 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 15:53:55.003237 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 15:53:55.004223 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:54.988099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:33381 (local address 127.13.223.3:35891)
0806 15:53:54.988669 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:54.988713 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:54.991116 (+ 2403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:54.991703 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:54.991714 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:54.992490 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:54.993587 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:54.993620 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.994481 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.994485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:54.995537 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:54.995596 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:54.996296 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:54.996300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:54.996535 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:54.997233 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:54.997249 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:54.998870 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:55.001252 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:55.001256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:55.001268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:55.001523 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:55.001987 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:55.001990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:55.001991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:55.002195 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:55.002671 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:55.002676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:55.002895 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 15:53:55.003057 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 15:53:55.003305 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":118,"threads_started":1}
I20260806 15:53:55.005417 15024 heartbeater.cc:347] Connected to a master server at 127.13.223.62:33381
I20260806 15:53:55.005710 15024 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:55.006211 15024 heartbeater.cc:511] Master 127.13.223.62:33381 requested a full tablet report, sending...
I20260806 15:53:55.007238 14285 ts_manager.cc:194] Registered new tserver with Master: c19e6f1602494afc85b5ebd9cdf74a14 (127.13.223.3:33957)
I20260806 15:53:55.007812 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:35891
I20260806 15:53:55.010175 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:53:55.039815 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:55.029579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.13.223.1:44641)
0806 15:53:55.029894 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:55.029904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:55.029934 (+ 30us) server_negotiation.cc:407] Connection header received
0806 15:53:55.030116 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:55.030121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:55.030309 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:55.030467 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:55.031184 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:55.031941 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:55.032901 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:55.033266 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:55.033384 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:55.035821 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:55.035834 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:55.035850 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:55.035896 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:55.037878 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:55.038375 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:55.038381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:55.038383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:55.038477 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:55.038771 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:55.038774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:55.038775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:55.039080 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:55.039247 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:55.039508 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 15:53:55.039662 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
W20260806 15:53:55.040951 14445 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51566
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[14214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[14214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:53:55.070432 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:55.061864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.13.223.1:44641)
0806 15:53:55.062015 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:55.062025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:55.062253 (+ 228us) server_negotiation.cc:407] Connection header received
0806 15:53:55.062426 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:55.062430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:55.062482 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:55.062567 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:55.063282 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:55.063826 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:55.064730 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:55.064956 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:55.065060 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:55.067502 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:55.067505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:55.067507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:55.067530 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:55.069049 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:55.069429 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:55.069432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:55.069433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:55.069504 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:55.069755 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:55.069757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:55.069758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:55.069912 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:55.070013 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:55.070212 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 15:53:55.070310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:53:55.071568 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14389
I20260806 15:53:55.087195 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14599
I20260806 15:53:55.099916 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14813
I20260806 15:53:55.109197 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14231
2026-08-06T15:53:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2870 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[15047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[15047](info): set up 2 sockets
Aug 06 15:53:55 dist-test-slave-rwrg krb5kdc[15047](info): commencing operation
krb5kdc: starting...
W20260806 15:53:56.382462 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:53:56 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:53:56.507942 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42957
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:56.620633 15070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:56.620916 15070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:56.620954 15070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:56.624815 15070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:53:56.624869 15070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:56.624889 15070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:56.624907 15070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:53:56.624922 15070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:53:56.630041 15070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:56.631042 15070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:56.632015 15070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:56.640475 15077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:56.640525 15075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:56.640525 15079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:56.640923 15070 server_base.cc:1034] running on GCE node
I20260806 15:53:56.641525 15070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:56.642621 15070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:56.643788 15070 hybrid_clock.cc:648] HybridClock initialized: now 1786031636643760 us; error 44 us; skew 500 ppm
Aug 06 15:53:56 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031636, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:56.652017 15070 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:53:56.653136 15070 webserver.cc:460] Webserver started at http://127.13.223.62:38593/ using document root <none> and password file <none>
I20260806 15:53:56.653731 15070 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:56.653795 15070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:56.654016 15070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:56.656091 15070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "210eac45d3f64f6f89730063c724bba8"
format_stamp: "Formatted at 2026-08-06 15:53:56 on dist-test-slave-rwrg"
server_key: "9686c701267baaac7ccaa78a7d72aa9c"
server_key_iv: "f60f570d6e7c69a4dd9d233bfd200acf"
server_key_version: "encryptionkey@0"
I20260806 15:53:56.656708 15070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "210eac45d3f64f6f89730063c724bba8"
format_stamp: "Formatted at 2026-08-06 15:53:56 on dist-test-slave-rwrg"
server_key: "9686c701267baaac7ccaa78a7d72aa9c"
server_key_iv: "f60f570d6e7c69a4dd9d233bfd200acf"
server_key_version: "encryptionkey@0"
I20260806 15:53:56.660583 15070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:53:56.662990 15089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:56.664268 15070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:53:56.664428 15070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "210eac45d3f64f6f89730063c724bba8"
format_stamp: "Formatted at 2026-08-06 15:53:56 on dist-test-slave-rwrg"
server_key: "9686c701267baaac7ccaa78a7d72aa9c"
server_key_iv: "f60f570d6e7c69a4dd9d233bfd200acf"
server_key_version: "encryptionkey@0"
I20260806 15:53:56.664579 15070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:56.673400 15070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:56.676241 15070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:56.676456 15070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:56.683836 15070 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:42957
I20260806 15:53:56.683854 15176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:42957 every 8 connection(s)
I20260806 15:53:56.684829 15070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:53:56.685384 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15070
I20260806 15:53:56.685531 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:53:56.685828 14204 external_mini_cluster.cc:1428] Setting key bcaced2b0c51808656e08da0575880b6
I20260806 15:53:56.688050 15177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:53:56.693524 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8: Bootstrap starting.
Aug 06 15:53:56 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:56.695812 15177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8: Neither blocks nor log segments found. Creating new log.
I20260806 15:53:56.696683 15177 log.cc:824] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8: Log is configured to *not* fsync() on all Append() calls
I20260806 15:53:56.698607 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8: No bootstrap required, opened a new log
I20260806 15:53:56.699956 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:56.687249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.13.223.62:42957)
0806 15:53:56.687694 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:56.687709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:56.687744 (+ 35us) server_negotiation.cc:407] Connection header received
0806 15:53:56.688424 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:56.688442 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:56.688826 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:56.689166 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:56.689984 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:56.690986 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:56.691839 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:56.692305 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:56.692479 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:56.695008 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:56.695013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:56.695025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:56.695050 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:56.696944 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:56.697391 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:56.697398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:56.697403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:56.697522 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:56.697821 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:56.697824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:56.697826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:56.698214 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:56.698420 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:56.698705 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 15:53:56.698985 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":147,"threads_started":1}
I20260806 15:53:56.701673 15177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210eac45d3f64f6f89730063c724bba8" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42957 } }
I20260806 15:53:56.701900 15177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:53:56.701972 15177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210eac45d3f64f6f89730063c724bba8, State: Initialized, Role: FOLLOWER
I20260806 15:53:56.702438 15177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210eac45d3f64f6f89730063c724bba8" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42957 } }
I20260806 15:53:56.702590 15177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:53:56.702677 15177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:53:56.702791 15177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:53:56.703668 15177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210eac45d3f64f6f89730063c724bba8" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42957 } }
I20260806 15:53:56.704022 15177 leader_election.cc:302] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210eac45d3f64f6f89730063c724bba8; no voters:
I20260806 15:53:56.704370 15177 leader_election.cc:288] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:53:56.704476 15186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:53:56.704725 15186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [term 1 LEADER]: Becoming Leader. State: Replica: 210eac45d3f64f6f89730063c724bba8, State: Running, Role: LEADER
I20260806 15:53:56.705186 15186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210eac45d3f64f6f89730063c724bba8" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42957 } }
I20260806 15:53:56.705634 15177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:53:56.706722 15187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210eac45d3f64f6f89730063c724bba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210eac45d3f64f6f89730063c724bba8" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42957 } } }
I20260806 15:53:56.706846 15187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [sys.catalog]: This master's current role is: LEADER
I20260806 15:53:56.707163 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210eac45d3f64f6f89730063c724bba8. Latest consensus state: current_term: 1 leader_uuid: "210eac45d3f64f6f89730063c724bba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210eac45d3f64f6f89730063c724bba8" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42957 } } }
I20260806 15:53:56.707270 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8 [sys.catalog]: This master's current role is: LEADER
I20260806 15:53:56.707352 15204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:53:56.710638 15204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:53:56.716207 15204 catalog_manager.cc:1347] Generated new cluster ID: 7c89ed4cfb584b849173cb5122492c9d
I20260806 15:53:56.716270 15204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:53:56.728838 15204 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:53:56.729769 15204 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:53:56.735577 15204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210eac45d3f64f6f89730063c724bba8: Generated new TSK 0
I20260806 15:53:56.736315 15204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:53:56.768431 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42957
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:56.881480 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:56.881776 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:56.881850 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:56.885864 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:56.885939 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:56.886029 15221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:53:56.891278 15221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:56.892406 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:56.893440 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:56.903443 15234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:56.903432 15231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:56.903417 15230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:56.903666 15221 server_base.cc:1034] running on GCE node
I20260806 15:53:56.904189 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:56.904908 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:56.906102 15221 hybrid_clock.cc:648] HybridClock initialized: now 1786031636906066 us; error 48 us; skew 500 ppm
Aug 06 15:53:56 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031636, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:56.908849 15221 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:53:56.909747 15221 webserver.cc:460] Webserver started at http://127.13.223.1:36833/ using document root <none> and password file <none>
I20260806 15:53:56.910252 15221 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:56.910310 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:56.910465 15221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:56.912382 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "e2425c74de884748858cd0162821466d"
format_stamp: "Formatted at 2026-08-06 15:53:56 on dist-test-slave-rwrg"
server_key: "f96c24ab3cb75fd975887f5fe13fc923"
server_key_iv: "f1b06fbd9aadde28d500e4f8576ec6d2"
server_key_version: "encryptionkey@0"
I20260806 15:53:56.912954 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "e2425c74de884748858cd0162821466d"
format_stamp: "Formatted at 2026-08-06 15:53:56 on dist-test-slave-rwrg"
server_key: "f96c24ab3cb75fd975887f5fe13fc923"
server_key_iv: "f1b06fbd9aadde28d500e4f8576ec6d2"
server_key_version: "encryptionkey@0"
I20260806 15:53:56.916563 15221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:53:56.918855 15246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:56.920042 15221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:53:56.920163 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "e2425c74de884748858cd0162821466d"
format_stamp: "Formatted at 2026-08-06 15:53:56 on dist-test-slave-rwrg"
server_key: "f96c24ab3cb75fd975887f5fe13fc923"
server_key_iv: "f1b06fbd9aadde28d500e4f8576ec6d2"
server_key_version: "encryptionkey@0"
I20260806 15:53:56.920265 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:56.943151 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:56.946272 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:56.946475 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:56.947088 15221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:53:56.947942 15221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:56.947989 15221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:56.948029 15221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:56.948076 15221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:56.957170 15221 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:42501
I20260806 15:53:56.957190 15446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:42501 every 8 connection(s)
I20260806 15:53:56.958063 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:53:56.965442 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15221
I20260806 15:53:56.965653 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:53:56.965931 14204 external_mini_cluster.cc:1428] Setting key d3460e81169d75f35fa25575cb15e309
Aug 06 15:53:56 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031636, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:56.972993 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:56.960019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:51799 (local address 127.13.223.62:42957)
0806 15:53:56.960169 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:56.960172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:56.961068 (+ 896us) server_negotiation.cc:407] Connection header received
0806 15:53:56.962396 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:56.962400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:56.962457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:56.962561 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:56.963748 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:56.964315 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:56.965524 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:56.965840 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:56.965951 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:56.968553 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:56.968556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:56.968558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:56.968580 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:56.970385 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:56.970933 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:56.970939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:56.970943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:56.971034 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:56.971445 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:56.971452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:56.971457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:56.971784 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:56.971895 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:56.972569 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 15:53:56.972739 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:53:56.973753 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:56.960394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42957 (local address 127.13.223.1:51799)
0806 15:53:56.960918 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:56.960961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:56.962186 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:56.962767 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:56.962774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:56.963126 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:56.963589 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:56.963601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:56.964452 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:56.964456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:56.965394 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:56.965403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:56.965995 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:56.965999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:56.966174 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:56.966799 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:56.966816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:56.968365 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:56.970553 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:56.970558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:56.970570 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:56.970826 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:56.971173 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:56.971175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:56.971177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:56.971338 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:56.971985 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:56.971991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:56.972244 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 15:53:56.972418 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 15:53:56.972783 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":124,"threads_started":1}
I20260806 15:53:56.975104 15447 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42957
I20260806 15:53:56.975345 15447 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:56.975878 15447 heartbeater.cc:511] Master 127.13.223.62:42957 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:53:56.977494 15121 ts_manager.cc:194] Registered new tserver with Master: e2425c74de884748858cd0162821466d (127.13.223.1:42501)
I20260806 15:53:56.978667 15121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:51799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:53:56.995600 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42957
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:57.111209 15457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:57.111492 15457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:57.111584 15457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:57.115681 15457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:57.115782 15457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:57.115875 15457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:53:57.121323 15457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:57.122395 15457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:57.123498 15457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:57.133553 15474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:57.133554 15472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:57.133673 15469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:57.134106 15457 server_base.cc:1034] running on GCE node
I20260806 15:53:57.134559 15457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:57.135309 15457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:57.136583 15457 hybrid_clock.cc:648] HybridClock initialized: now 1786031637136464 us; error 52 us; skew 500 ppm
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:57.139492 15457 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:53:57.140599 15457 webserver.cc:460] Webserver started at http://127.13.223.2:36479/ using document root <none> and password file <none>
I20260806 15:53:57.141215 15457 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:57.141288 15457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:57.141553 15457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:57.143743 15457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "e390b050b2484b069af0063a75604e6e"
format_stamp: "Formatted at 2026-08-06 15:53:57 on dist-test-slave-rwrg"
server_key: "b2438fe2b082c1b3b5f704e4a0753d7b"
server_key_iv: "238431570bfeb6cca611e288ea7b6d23"
server_key_version: "encryptionkey@0"
I20260806 15:53:57.144371 15457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "e390b050b2484b069af0063a75604e6e"
format_stamp: "Formatted at 2026-08-06 15:53:57 on dist-test-slave-rwrg"
server_key: "b2438fe2b082c1b3b5f704e4a0753d7b"
server_key_iv: "238431570bfeb6cca611e288ea7b6d23"
server_key_version: "encryptionkey@0"
I20260806 15:53:57.148284 15457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 15:53:57.150983 15485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:57.152246 15457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:53:57.152405 15457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "e390b050b2484b069af0063a75604e6e"
format_stamp: "Formatted at 2026-08-06 15:53:57 on dist-test-slave-rwrg"
server_key: "b2438fe2b082c1b3b5f704e4a0753d7b"
server_key_iv: "238431570bfeb6cca611e288ea7b6d23"
server_key_version: "encryptionkey@0"
I20260806 15:53:57.152590 15457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:57.163164 15457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:57.166041 15457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:57.166242 15457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:57.166836 15457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:53:57.167665 15457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:57.167711 15457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:57.167811 15457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:57.167852 15457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:57.181237 15457 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:36057
I20260806 15:53:57.181258 15673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:36057 every 8 connection(s)
I20260806 15:53:57.182217 15457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:53:57.182451 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15457
I20260806 15:53:57.182567 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:53:57.182834 14204 external_mini_cluster.cc:1428] Setting key 9869a5c89aa8eb999fdd2ece8a5f1751
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:53:57.199996 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.184394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:44331 (local address 127.13.223.62:42957)
0806 15:53:57.184559 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.184563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.187719 (+ 3156us) server_negotiation.cc:407] Connection header received
0806 15:53:57.189063 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.189065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.189120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.189211 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:57.190964 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.191478 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.193383 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.193610 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.193711 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.196524 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:57.196528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:57.196531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:57.196558 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:57.197905 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.198441 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.198445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.198447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.198541 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.198890 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.198895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.198897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.199083 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:57.199196 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.199707 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.199830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 15:53:57.201198 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.184905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42957 (local address 127.13.223.2:44331)
0806 15:53:57.187564 (+ 2659us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.187609 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.188867 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.189451 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.189462 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.190032 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:57.190786 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.190811 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.191621 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.191628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.193258 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.193270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.193783 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.193787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.194007 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.194628 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:57.194655 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:57.196304 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:57.198028 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:57.198033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:57.198067 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:57.198333 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:57.198640 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:57.198643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:57.198645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:57.198788 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:57.199196 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:57.199204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:57.199452 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.199664 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.199981 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2467,"thread_start_us":119,"threads_started":1}
I20260806 15:53:57.202443 15674 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42957
I20260806 15:53:57.202719 15674 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:57.203233 15674 heartbeater.cc:511] Master 127.13.223.62:42957 requested a full tablet report, sending...
I20260806 15:53:57.204366 15121 ts_manager.cc:194] Registered new tserver with Master: e390b050b2484b069af0063a75604e6e (127.13.223.2:36057)
I20260806 15:53:57.204977 15121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:44331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:53:57.216091 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42957
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:57.337877 15681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:57.338131 15681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:57.338212 15681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:57.342262 15681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:57.342324 15681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:57.342425 15681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:53:57.347885 15681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:57.349094 15681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:57.350211 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:57.363139 15688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:57.363139 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:57.363152 15691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:57.363622 15681 server_base.cc:1034] running on GCE node
I20260806 15:53:57.363972 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:57.364615 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:57.365784 15681 hybrid_clock.cc:648] HybridClock initialized: now 1786031637365770 us; error 25 us; skew 500 ppm
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:57.368706 15681 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:53:57.369665 15681 webserver.cc:460] Webserver started at http://127.13.223.3:39715/ using document root <none> and password file <none>
I20260806 15:53:57.370234 15681 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:57.370296 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:57.370532 15681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:57.372637 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "cb4af7b458ad44b9a634972087b49e33"
format_stamp: "Formatted at 2026-08-06 15:53:57 on dist-test-slave-rwrg"
server_key: "7263e2edb3170690c1fce4cd580b1982"
server_key_iv: "b3b4986fe3e2c260669560090129b12b"
server_key_version: "encryptionkey@0"
I20260806 15:53:57.373199 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "cb4af7b458ad44b9a634972087b49e33"
format_stamp: "Formatted at 2026-08-06 15:53:57 on dist-test-slave-rwrg"
server_key: "7263e2edb3170690c1fce4cd580b1982"
server_key_iv: "b3b4986fe3e2c260669560090129b12b"
server_key_version: "encryptionkey@0"
I20260806 15:53:57.376958 15681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:53:57.379385 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:57.380681 15681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:53:57.380797 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "cb4af7b458ad44b9a634972087b49e33"
format_stamp: "Formatted at 2026-08-06 15:53:57 on dist-test-slave-rwrg"
server_key: "7263e2edb3170690c1fce4cd580b1982"
server_key_iv: "b3b4986fe3e2c260669560090129b12b"
server_key_version: "encryptionkey@0"
I20260806 15:53:57.380897 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:57.393286 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:57.397116 15681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:57.397375 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:57.398262 15681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:53:57.399261 15681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:57.399313 15681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:57.399358 15681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:57.399379 15681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:57.415359 15681 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:36803
I20260806 15:53:57.415377 15894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:36803 every 8 connection(s)
I20260806 15:53:57.416599 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:53:57.424299 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15681
I20260806 15:53:57.424441 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:53:57.424757 14204 external_mini_cluster.cc:1428] Setting key 5849c8c7993d2cbaebd6cee7722133a8
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:57.432289 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.418639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:46441 (local address 127.13.223.62:42957)
0806 15:53:57.418788 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.418792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.419746 (+ 954us) server_negotiation.cc:407] Connection header received
0806 15:53:57.421200 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.421203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.421264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.421340 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:57.422663 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.423173 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.424851 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.425166 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.425290 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.428723 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:57.428727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:57.428730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:57.428756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:57.430271 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.430778 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.430781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.430782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.430855 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.431236 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.431239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.431240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.431412 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:57.431486 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.432064 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.432172 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:53:57.433285 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.419014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42957 (local address 127.13.223.3:46441)
0806 15:53:57.419506 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.419598 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.420992 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.421516 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.421560 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.421937 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:57.422518 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.422543 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.423300 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.423303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.424715 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.424726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.425340 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.425344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.425634 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.426463 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:57.426488 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:57.428455 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:57.430383 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:57.430387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:57.430399 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:57.430678 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:57.430960 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:57.430963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:57.430965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:57.431131 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:57.431545 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:57.431550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:57.431820 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.431997 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.432272 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":155,"threads_started":1}
I20260806 15:53:57.434477 15895 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42957
I20260806 15:53:57.434758 15895 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:57.435220 15895 heartbeater.cc:511] Master 127.13.223.62:42957 requested a full tablet report, sending...
I20260806 15:53:57.436241 15121 ts_manager.cc:194] Registered new tserver with Master: cb4af7b458ad44b9a634972087b49e33 (127.13.223.3:36803)
I20260806 15:53:57.436961 15121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:46441
I20260806 15:53:57.439054 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:53:57.450675 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.442983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.13.223.62:42957)
0806 15:53:57.443118 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.443121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.443333 (+ 212us) server_negotiation.cc:407] Connection header received
0806 15:53:57.443427 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.443430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.443466 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.443533 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:57.444229 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.444802 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.445686 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.445883 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.445957 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.447371 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:57.447373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:57.447375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:57.447397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:57.448985 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.449440 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.449443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.449444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.449553 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.449851 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.449853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.449855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.450065 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:57.450189 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.450453 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.450545 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:53:57.456525 15121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:53:57.460022 15121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:53:57.476565 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.470321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42501 (local address 127.0.0.1:44400)
0806 15:53:57.470746 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.470797 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.470955 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.472154 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.472160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.472183 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:53:57.472340 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.472347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.473663 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.473669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.474808 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.474817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.476001 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.476004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.476117 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.476157 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.476267 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.476375 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260806 15:53:57.476601 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.470498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.13.223.1:42501)
0806 15:53:57.471617 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.471647 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.471684 (+ 37us) server_negotiation.cc:407] Connection header received
0806 15:53:57.471771 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.471777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.472003 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.472165 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:53:57.472457 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.473534 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.474970 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.475863 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.475985 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.476177 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.476295 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.476388 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":156,"threads_started":1}
I20260806 15:53:57.477635 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.470572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:36057 (local address 127.0.0.1:57444)
0806 15:53:57.471232 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.471245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.471344 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.472597 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.472600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.472620 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:53:57.472779 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.472787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.474713 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.474716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.475934 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.475947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.477214 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.477216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.477289 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.477306 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.477411 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.477484 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":101,"threads_started":1}
I20260806 15:53:57.477840 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.471431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.13.223.2:36057)
0806 15:53:57.471985 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.472041 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.472066 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:53:57.472167 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.472172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.472382 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.472628 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:53:57.472907 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.474511 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.476078 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.476908 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.477033 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.477159 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.477458 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.477569 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":274,"threads_started":1}
I20260806 15:53:57.478788 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.471195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:36803 (local address 127.0.0.1:38302)
0806 15:53:57.472452 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.472466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.472613 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.473251 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.473254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.473273 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:53:57.473348 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.473353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.475436 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.475443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.476448 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.476457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.478386 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.478390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.478474 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.478496 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.478587 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.478651 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":153,"threads_started":1}
I20260806 15:53:57.479332 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.471430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.13.223.3:36803)
0806 15:53:57.472220 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.472265 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.472723 (+ 458us) server_negotiation.cc:407] Connection header received
0806 15:53:57.472836 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.472841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.473062 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.473260 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:53:57.473493 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.475221 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.476606 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.477705 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.478195 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.478329 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.478954 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.479067 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":185,"threads_started":1}
I20260806 15:53:57.480573 15566 tablet_service.cc:1467] Processing CreateTablet for tablet d579e518437f467a882512db754caaa2 (DEFAULT_TABLE table=test-table [id=d9e87c977ceb4eb7a2a417cb2d5f5fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:53:57.480554 15329 tablet_service.cc:1467] Processing CreateTablet for tablet d579e518437f467a882512db754caaa2 (DEFAULT_TABLE table=test-table [id=d9e87c977ceb4eb7a2a417cb2d5f5fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:53:57.481612 15566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d579e518437f467a882512db754caaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:53:57.481662 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d579e518437f467a882512db754caaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:53:57.482151 15788 tablet_service.cc:1467] Processing CreateTablet for tablet d579e518437f467a882512db754caaa2 (DEFAULT_TABLE table=test-table [id=d9e87c977ceb4eb7a2a417cb2d5f5fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:53:57.483209 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d579e518437f467a882512db754caaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:53:57.489817 15927 tablet_bootstrap.cc:492] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: Bootstrap starting.
I20260806 15:53:57.491340 15928 tablet_bootstrap.cc:492] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: Bootstrap starting.
I20260806 15:53:57.492695 15927 tablet_bootstrap.cc:654] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: Neither blocks nor log segments found. Creating new log.
I20260806 15:53:57.493587 15927 log.cc:824] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: Log is configured to *not* fsync() on all Append() calls
I20260806 15:53:57.494280 15928 tablet_bootstrap.cc:654] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: Neither blocks nor log segments found. Creating new log.
I20260806 15:53:57.495131 15928 log.cc:824] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: Log is configured to *not* fsync() on all Append() calls
I20260806 15:53:57.495942 15932 tablet_bootstrap.cc:492] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: Bootstrap starting.
I20260806 15:53:57.498507 15928 tablet_bootstrap.cc:492] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: No bootstrap required, opened a new log
I20260806 15:53:57.498705 15927 tablet_bootstrap.cc:492] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: No bootstrap required, opened a new log
I20260806 15:53:57.498735 15928 ts_tablet_manager.cc:1397] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260806 15:53:57.498942 15932 tablet_bootstrap.cc:654] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: Neither blocks nor log segments found. Creating new log.
I20260806 15:53:57.499092 15927 ts_tablet_manager.cc:1397] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260806 15:53:57.501940 15932 log.cc:824] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: Log is configured to *not* fsync() on all Append() calls
I20260806 15:53:57.504077 15927 raft_consensus.cc:348] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.504400 15927 raft_consensus.cc:374] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:53:57.504468 15927 raft_consensus.cc:729] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e390b050b2484b069af0063a75604e6e, State: Initialized, Role: FOLLOWER
I20260806 15:53:57.505069 15927 consensus_queue.cc:260] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.506125 15674 heartbeater.cc:503] Master 127.13.223.62:42957 was elected leader, sending a full tablet report...
I20260806 15:53:57.506289 15927 ts_tablet_manager.cc:1428] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260806 15:53:57.506659 15932 tablet_bootstrap.cc:492] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: No bootstrap required, opened a new log
I20260806 15:53:57.506851 15932 ts_tablet_manager.cc:1397] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 15:53:57.508623 15928 raft_consensus.cc:348] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.508910 15928 raft_consensus.cc:374] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:53:57.508960 15928 raft_consensus.cc:729] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2425c74de884748858cd0162821466d, State: Initialized, Role: FOLLOWER
I20260806 15:53:57.509513 15928 consensus_queue.cc:260] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.510692 15928 ts_tablet_manager.cc:1428] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260806 15:53:57.511008 15447 heartbeater.cc:503] Master 127.13.223.62:42957 was elected leader, sending a full tablet report...
I20260806 15:53:57.513101 15932 raft_consensus.cc:348] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.513384 15932 raft_consensus.cc:374] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:53:57.513442 15932 raft_consensus.cc:729] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4af7b458ad44b9a634972087b49e33, State: Initialized, Role: FOLLOWER
I20260806 15:53:57.513888 15932 consensus_queue.cc:260] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.514891 15932 ts_tablet_manager.cc:1428] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260806 15:53:57.514964 15895 heartbeater.cc:503] Master 127.13.223.62:42957 was elected leader, sending a full tablet report...
I20260806 15:53:57.634279 15935 raft_consensus.cc:482] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:53:57.634608 15935 raft_consensus.cc:504] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.635929 15935 leader_election.cc:288] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e390b050b2484b069af0063a75604e6e (127.13.223.2:36057), cb4af7b458ad44b9a634972087b49e33 (127.13.223.3:36803)
I20260806 15:53:57.643127 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.636578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:50599 (local address 127.13.223.3:36803)
0806 15:53:57.636828 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.636836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.637443 (+ 607us) server_negotiation.cc:407] Connection header received
0806 15:53:57.637509 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.637512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.637565 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.637658 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:53:57.638340 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.639130 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.641536 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.642286 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.642397 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.642449 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.642912 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.642984 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 15:53:57.643119 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.636609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:37453 (local address 127.13.223.2:36057)
0806 15:53:57.636843 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.636849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.637295 (+ 446us) server_negotiation.cc:407] Connection header received
0806 15:53:57.637511 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.637514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.637568 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.637679 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:53:57.638340 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.639154 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.641536 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.642258 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.642398 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.642449 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.642912 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.642984 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:53:57.643241 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.636382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:36803 (local address 127.13.223.1:50599)
0806 15:53:57.637267 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.637284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.637417 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.637700 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.637703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.637748 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:53:57.638210 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.638220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.639269 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.639273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.641415 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.641427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.642434 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.642439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.642658 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.642774 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.642866 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.642977 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":150,"spinlock_wait_cycles":9344,"thread_start_us":474,"threads_started":1}
I20260806 15:53:57.643265 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.636321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:36057 (local address 127.13.223.1:37453)
0806 15:53:57.637125 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:57.637144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:57.637360 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:57.637701 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:57.637704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:57.637747 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:53:57.638165 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.638192 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.639269 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.639273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.641379 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:57.641393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.642435 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.642439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:57.642747 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.642788 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 15:53:57.642866 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 15:53:57.642977 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":262,"spinlock_wait_cycles":33664,"thread_start_us":484,"threads_started":1}
I20260806 15:53:57.644325 15821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579e518437f467a882512db754caaa2" candidate_uuid: "e2425c74de884748858cd0162821466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4af7b458ad44b9a634972087b49e33" is_pre_election: true
I20260806 15:53:57.644440 15595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579e518437f467a882512db754caaa2" candidate_uuid: "e2425c74de884748858cd0162821466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e390b050b2484b069af0063a75604e6e" is_pre_election: true
I20260806 15:53:57.644780 15821 raft_consensus.cc:2454] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2425c74de884748858cd0162821466d in term 0.
I20260806 15:53:57.644892 15595 raft_consensus.cc:2454] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2425c74de884748858cd0162821466d in term 0.
I20260806 15:53:57.645423 15250 leader_election.cc:302] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb4af7b458ad44b9a634972087b49e33, e2425c74de884748858cd0162821466d; no voters:
I20260806 15:53:57.645742 15935 raft_consensus.cc:2781] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:53:57.645849 15935 raft_consensus.cc:482] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:53:57.645896 15935 raft_consensus.cc:3037] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:53:57.647576 15935 raft_consensus.cc:504] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.647957 15935 leader_election.cc:288] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [CANDIDATE]: Term 1 election: Requested vote from peers e390b050b2484b069af0063a75604e6e (127.13.223.2:36057), cb4af7b458ad44b9a634972087b49e33 (127.13.223.3:36803)
I20260806 15:53:57.648418 15595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579e518437f467a882512db754caaa2" candidate_uuid: "e2425c74de884748858cd0162821466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e390b050b2484b069af0063a75604e6e"
I20260806 15:53:57.648542 15821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579e518437f467a882512db754caaa2" candidate_uuid: "e2425c74de884748858cd0162821466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4af7b458ad44b9a634972087b49e33"
I20260806 15:53:57.648604 15595 raft_consensus.cc:3037] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:53:57.648711 15821 raft_consensus.cc:3037] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:53:57.650063 15595 raft_consensus.cc:2454] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2425c74de884748858cd0162821466d in term 1.
I20260806 15:53:57.650324 15821 raft_consensus.cc:2454] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2425c74de884748858cd0162821466d in term 1.
I20260806 15:53:57.650513 15247 leader_election.cc:302] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2425c74de884748858cd0162821466d, e390b050b2484b069af0063a75604e6e; no voters:
I20260806 15:53:57.650713 15935 raft_consensus.cc:2781] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:53:57.650936 15935 raft_consensus.cc:686] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [term 1 LEADER]: Becoming Leader. State: Replica: e2425c74de884748858cd0162821466d, State: Running, Role: LEADER
I20260806 15:53:57.651320 15935 consensus_queue.cc:237] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } }
I20260806 15:53:57.654883 15119 catalog_manager.cc:5495] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d reported cstate change: term changed from 0 to 1, leader changed from <none> to e2425c74de884748858cd0162821466d (127.13.223.1). New cstate: current_term: 1 leader_uuid: "e2425c74de884748858cd0162821466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2425c74de884748858cd0162821466d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e390b050b2484b069af0063a75604e6e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb4af7b458ad44b9a634972087b49e33" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 36803 } health_report { overall_health: UNKNOWN } } }
W20260806 15:53:57.667995 15898 tablet.cc:2369] T d579e518437f467a882512db754caaa2 P cb4af7b458ad44b9a634972087b49e33: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:53:57.680693 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.670702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.13.223.1:42501)
0806 15:53:57.670886 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.670891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.671077 (+ 186us) server_negotiation.cc:407] Connection header received
0806 15:53:57.671183 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.671185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.671234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.671297 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:57.672177 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.672886 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.673932 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.674167 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.674255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.676981 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:57.676994 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:57.677011 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:57.677037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:57.678807 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.679261 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.679268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.679270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.679376 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.679623 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.679625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.679626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.679916 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:57.680118 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.680334 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.680468 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 15:53:57.683760 15676 tablet.cc:2369] T d579e518437f467a882512db754caaa2 P e390b050b2484b069af0063a75604e6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260806 15:53:57.709679 15448 tablet.cc:2369] T d579e518437f467a882512db754caaa2 P e2425c74de884748858cd0162821466d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:53:57.716071 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.704190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.13.223.1:42501)
0806 15:53:57.704332 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.704337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.704524 (+ 187us) server_negotiation.cc:407] Connection header received
0806 15:53:57.704738 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.704745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.704809 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.704888 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:57.705995 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.707110 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.708575 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.709099 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.709230 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.712362 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:57.712382 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:57.712385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:57.712415 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:57.714319 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.714725 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.714729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.714731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.714816 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.715135 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.715138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.715140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.715369 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:57.715510 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.715800 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.715921 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:53:57.746281 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:57.736929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.13.223.1:42501)
0806 15:53:57.737078 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:57.737082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:57.737266 (+ 184us) server_negotiation.cc:407] Connection header received
0806 15:53:57.737425 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:57.737429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:57.737478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:57.737572 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:57.738252 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.738807 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.739797 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:57.740002 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:57.740083 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:57.742728 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:57.742732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:57.742734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:57.742761 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:57.744351 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.744865 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.744869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.744871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.744968 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:57.745279 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:57.745284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:57.745286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:57.745482 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:57.745802 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:57.745948 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 15:53:57.746091 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 15:53:57.747951 15287 tablet_service.cc:2908] Rejecting scan request for tablet d579e518437f467a882512db754caaa2: Uninitialized: clean time has not yet been initialized
I20260806 15:53:57.750604 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15221
I20260806 15:53:57.764422 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15457
I20260806 15:53:57.775385 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15681
I20260806 15:53:57.786104 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15070
2026-08-06T15:53:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2679 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15975](info): set up 2 sockets
Aug 06 15:53:57 dist-test-slave-rwrg krb5kdc[15975](info): commencing operation
krb5kdc: starting...
W20260806 15:53:59.072840 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:53:59.200572 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39443
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:59.311872 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:59.312153 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:59.312239 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:59.316246 15996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:53:59.316360 15996 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 15:53:59.316391 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:59.316452 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:59.316520 15996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:53:59.316565 15996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:53:59.321841 15996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:39443
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:59.322891 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:59.323922 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:59.332719 16007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:59.332723 16006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:59.332796 16013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:59.333168 15996 server_base.cc:1034] running on GCE node
I20260806 15:53:59.333725 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:59.334823 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:59.336032 15996 hybrid_clock.cc:648] HybridClock initialized: now 1786031639336010 us; error 28 us; skew 500 ppm
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:59.338927 15996 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:53:59.339785 15996 webserver.cc:460] Webserver started at http://127.13.223.62:34829/ using document root <none> and password file <none>
I20260806 15:53:59.340301 15996 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:59.340364 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:59.340610 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:59.342746 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "f8ef6559ad6c485b9be186644e032dd3"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "234cee2a315ba8573eb579166a67d248"
server_key_iv: "4109e1340b15f5feefc90edbd5bd55c6"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.343266 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "f8ef6559ad6c485b9be186644e032dd3"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "234cee2a315ba8573eb579166a67d248"
server_key_iv: "4109e1340b15f5feefc90edbd5bd55c6"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.347260 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:53:59.349735 16024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:59.350976 15996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:53:59.351089 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "f8ef6559ad6c485b9be186644e032dd3"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "234cee2a315ba8573eb579166a67d248"
server_key_iv: "4109e1340b15f5feefc90edbd5bd55c6"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.351181 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:59.365912 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:59.368929 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:59.369081 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:59.378175 15996 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:39443
I20260806 15:53:59.378207 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:39443 every 8 connection(s)
I20260806 15:53:59.379199 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:53:59.382903 16119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:53:59.387399 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15996
I20260806 15:53:59.387519 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:53:59.387780 14204 external_mini_cluster.cc:1428] Setting key 0966c4001b71827d149f533c404df862
I20260806 15:53:59.389396 16119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: Bootstrap starting.
I20260806 15:53:59.391779 16119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: Neither blocks nor log segments found. Creating new log.
I20260806 15:53:59.392863 16119 log.cc:824] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: Log is configured to *not* fsync() on all Append() calls
I20260806 15:53:59.395275 16119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: No bootstrap required, opened a new log
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:59.399767 16119 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef6559ad6c485b9be186644e032dd3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39443 } }
I20260806 15:53:59.400066 16119 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:53:59.400161 16119 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ef6559ad6c485b9be186644e032dd3, State: Initialized, Role: FOLLOWER
I20260806 15:53:59.400754 16119 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef6559ad6c485b9be186644e032dd3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39443 } }
I20260806 15:53:59.400945 16119 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:53:59.401022 16119 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:53:59.401127 16119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:53:59.402457 16119 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef6559ad6c485b9be186644e032dd3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39443 } }
I20260806 15:53:59.402835 16119 leader_election.cc:302] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8ef6559ad6c485b9be186644e032dd3; no voters:
I20260806 15:53:59.403183 16119 leader_election.cc:288] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:53:59.403312 16129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:53:59.403327 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.389639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.13.223.62:39443)
0806 15:53:59.390128 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:59.390144 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:59.390182 (+ 38us) server_negotiation.cc:407] Connection header received
0806 15:53:59.390887 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:59.390905 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:59.391331 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:59.391697 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:59.392623 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.393670 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.394543 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.395201 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.395409 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.398004 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:59.398009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:59.398023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:59.398063 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:59.399769 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.400371 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.400378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.400383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.400565 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.400849 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.400853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.400855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.401432 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:59.401689 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:59.401899 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 15:53:59.402265 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":155,"threads_started":1}
I20260806 15:53:59.403651 16129 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [term 1 LEADER]: Becoming Leader. State: Replica: f8ef6559ad6c485b9be186644e032dd3, State: Running, Role: LEADER
I20260806 15:53:59.403986 16129 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef6559ad6c485b9be186644e032dd3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39443 } }
I20260806 15:53:59.404572 16119 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:53:59.405218 16129 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ef6559ad6c485b9be186644e032dd3. Latest consensus state: current_term: 1 leader_uuid: "f8ef6559ad6c485b9be186644e032dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef6559ad6c485b9be186644e032dd3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39443 } } }
I20260806 15:53:59.405318 16129 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [sys.catalog]: This master's current role is: LEADER
I20260806 15:53:59.405535 16134 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8ef6559ad6c485b9be186644e032dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef6559ad6c485b9be186644e032dd3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39443 } } }
I20260806 15:53:59.405613 16134 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3 [sys.catalog]: This master's current role is: LEADER
I20260806 15:53:59.405880 16138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:53:59.409346 16138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:53:59.414943 16138 catalog_manager.cc:1347] Generated new cluster ID: 7a4bdabb71f447ac8b3047e9865c7bb3
I20260806 15:53:59.415019 16138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:53:59.458302 16138 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:53:59.459287 16138 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:53:59.465549 16138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: Generated new TSK 0
I20260806 15:53:59.466243 16138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:53:59.507426 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39443
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:59.621994 16159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:59.622269 16159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:59.622347 16159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:59.626340 16159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:59.626416 16159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:59.626498 16159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:53:59.626579 16159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:53:59.631750 16159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:59.632830 16159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:59.633839 16159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:59.643392 16169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:59.643392 16166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:59.643399 16165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:59.643790 16159 server_base.cc:1034] running on GCE node
I20260806 15:53:59.644268 16159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:59.644955 16159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:59.646147 16159 hybrid_clock.cc:648] HybridClock initialized: now 1786031639646137 us; error 30 us; skew 500 ppm
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:59.648978 16159 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:53:59.649973 16159 webserver.cc:460] Webserver started at http://127.13.223.1:43641/ using document root <none> and password file <none>
I20260806 15:53:59.650533 16159 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:59.650661 16159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:59.650892 16159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:59.652993 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "cb55852d9cbfb225c7673e327a0b1973"
server_key_iv: "457376b2d302b9fb4fdbadae65eb7a32"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.653569 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "cb55852d9cbfb225c7673e327a0b1973"
server_key_iv: "457376b2d302b9fb4fdbadae65eb7a32"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.657573 16159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 15:53:59.660089 16184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:59.661391 16159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:53:59.661535 16159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "cb55852d9cbfb225c7673e327a0b1973"
server_key_iv: "457376b2d302b9fb4fdbadae65eb7a32"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.661669 16159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:59.681303 16159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:59.684557 16159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:59.684769 16159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:59.686247 16159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:59.686321 16159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:59.686394 16159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:59.686422 16159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:59.697906 16159 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:34413
I20260806 15:53:59.697990 16375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:34413 every 8 connection(s)
I20260806 15:53:59.699081 16159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:53:59.701563 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.686979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:51233 (local address 127.13.223.62:39443)
0806 15:53:59.687100 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:59.687104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:59.688927 (+ 1823us) server_negotiation.cc:407] Connection header received
0806 15:53:59.690088 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:59.690092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:59.690154 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:59.690258 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:59.691806 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.692458 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.693786 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.694226 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.694367 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.697340 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:59.697345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:59.697348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:59.697377 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:59.699296 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.699903 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.699906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.699908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.700001 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.700362 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.700366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.700368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.700637 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:59.700743 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:59.701247 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 15:53:59.701379 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 15:53:59.702307 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.688230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.13.223.1:51233)
0806 15:53:59.688771 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:59.688814 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:59.689875 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:59.690431 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:59.690437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:59.690949 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:59.691638 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.691674 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.692618 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.692621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.693650 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.693660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.694424 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.694430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.694622 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.695265 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:59.695280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:59.697119 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:59.699407 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.699411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.699422 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.699778 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.700126 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.700129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.700131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.700266 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.700752 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:59.700760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:59.700992 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 15:53:59.701190 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 15:53:59.701517 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":128,"threads_started":1}
I20260806 15:53:59.703482 16376 heartbeater.cc:347] Connected to a master server at 127.13.223.62:39443
I20260806 15:53:59.703656 16376 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:59.704181 16376 heartbeater.cc:511] Master 127.13.223.62:39443 requested a full tablet report, sending...
I20260806 15:53:59.706128 16044 ts_manager.cc:194] Registered new tserver with Master: 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413)
I20260806 15:53:59.708024 16044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:51233
I20260806 15:53:59.708681 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16159
I20260806 15:53:59.708809 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:53:59.709138 14204 external_mini_cluster.cc:1428] Setting key e17faf07b695980fed4d141850213359
I20260806 15:53:59.719586 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.710296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.13.223.1:54369)
0806 15:53:59.711060 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:59.711073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:59.711161 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:59.712024 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:59.712026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:59.712209 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:59.712308 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.712313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.713310 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.713317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.714260 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.714293 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.714941 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.714944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.715014 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.715556 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:59.715567 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:59.715924 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:59.718089 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.718091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.718093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.718342 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.718643 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.718645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.718646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.718722 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.719166 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:59.719168 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:59.719243 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 15:53:59.719325 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 15:53:59.719431 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":115,"threads_started":1}
I20260806 15:53:59.719735 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.710147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:54369 (local address 127.13.223.62:39443)
0806 15:53:59.711793 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:59.711800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:59.711817 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:53:59.711859 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:59.711864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:59.711916 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:59.712002 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:59.712431 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.713176 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.714416 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.714822 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.714945 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.716076 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:59.716081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:59.716086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:59.716116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:59.717966 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.718444 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.718450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.718452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.718544 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.718821 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.718826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.718829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.719072 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:59.719179 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:59.719415 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 15:53:59.719548 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544}
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:53:59.739997 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39443
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:53:59.853065 16393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:53:59.853338 16393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:53:59.853384 16393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:53:59.857395 16393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:53:59.857453 16393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:53:59.857491 16393 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:53:59.857523 16393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:53:59.862815 16393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:53:59.863816 16393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:53:59.864876 16393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:53:59.874756 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:59.874825 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:53:59.874756 16406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:53:59.874969 16393 server_base.cc:1034] running on GCE node
I20260806 15:53:59.875492 16393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:53:59.876161 16393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:53:59.877352 16393 hybrid_clock.cc:648] HybridClock initialized: now 1786031639877332 us; error 26 us; skew 500 ppm
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:53:59.880374 16393 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:53:59.881255 16393 webserver.cc:460] Webserver started at http://127.13.223.2:42153/ using document root <none> and password file <none>
I20260806 15:53:59.881811 16393 fs_manager.cc:359] Metadata directory not provided
I20260806 15:53:59.881904 16393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:53:59.882143 16393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:53:59.884130 16393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "003c245300614bde9658284dd52ef798"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "5514efc6d203737fc1c9fc6cdfd2df66"
server_key_iv: "a715e64c72fc2e303c15e515efab8328"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.884742 16393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "003c245300614bde9658284dd52ef798"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "5514efc6d203737fc1c9fc6cdfd2df66"
server_key_iv: "a715e64c72fc2e303c15e515efab8328"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.888471 16393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:53:59.891152 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:59.892444 16393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:53:59.892656 16393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "003c245300614bde9658284dd52ef798"
format_stamp: "Formatted at 2026-08-06 15:53:59 on dist-test-slave-rwrg"
server_key: "5514efc6d203737fc1c9fc6cdfd2df66"
server_key_iv: "a715e64c72fc2e303c15e515efab8328"
server_key_version: "encryptionkey@0"
I20260806 15:53:59.892781 16393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:53:59.905923 16393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:53:59.909371 16393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:53:59.909575 16393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:53:59.910975 16393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:53:59.911067 16393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:53:59.911161 16393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:53:59.911213 16393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:53:59 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:53:59.924142 16393 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:44575
I20260806 15:53:59.924162 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:44575 every 8 connection(s)
I20260806 15:53:59.925465 16393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:53:59.926352 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.912108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36447 (local address 127.13.223.62:39443)
0806 15:53:59.912247 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:59.912252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:59.913314 (+ 1062us) server_negotiation.cc:407] Connection header received
0806 15:53:59.914591 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:59.914594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:59.914653 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:59.914746 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:59.916456 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.917375 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.918664 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.918941 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.919064 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.921828 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:59.921832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:59.921835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:59.921866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:59.923336 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.924414 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.924418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.924420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.924563 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.924962 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.924965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.924966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.925173 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:59.925253 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:59.926011 (+ 758us) server_negotiation.cc:299] Negotiation successful
0806 15:53:59.926169 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:53:59.926570 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16393
I20260806 15:53:59.926697 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:53:59.927018 14204 external_mini_cluster.cc:1428] Setting key 7f3ec5ecf8295955ebe3d646f5f8f54c
I20260806 15:53:59.927234 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.912652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.13.223.2:36447)
0806 15:53:59.913140 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:59.913182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:59.914360 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:59.914949 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:59.914961 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:59.915481 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:59.916258 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.916281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.917507 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.917511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.918523 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.918535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.919125 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.919130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.919299 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.919926 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:59.919950 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:59.921650 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:59.923464 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.923470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.923487 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.924126 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.924669 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.924673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.924675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.924853 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.925283 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:59.925289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:59.925609 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 15:53:59.925875 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 15:53:59.926256 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260806 15:53:59.928740 16618 heartbeater.cc:347] Connected to a master server at 127.13.223.62:39443
I20260806 15:53:59.928926 16618 heartbeater.cc:464] Registering TS with master...
I20260806 15:53:59.929811 16618 heartbeater.cc:511] Master 127.13.223.62:39443 requested a full tablet report, sending...
I20260806 15:53:59.931048 16044 ts_manager.cc:194] Registered new tserver with Master: 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
I20260806 15:53:59.931843 16044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:36447
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:53:59.942432 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.930949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.13.223.2:57985)
0806 15:53:59.931357 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 15:53:59.931374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:53:59.931476 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:53:59.934637 (+ 3161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:53:59.934642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:53:59.934912 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:53:59.935081 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.935095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.935899 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.935904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.937074 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:53:59.937089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.937547 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.937561 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:53:59.937690 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.938234 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:53:59.938257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:53:59.938876 (+ 619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:53:59.940730 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.940737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.940742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.940986 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.941307 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:53:59.941309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:53:59.941310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:53:59.941392 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:53:59.941846 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:53:59.941848 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:53:59.941928 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 15:53:59.942016 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 15:53:59.942200 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":228,"threads_started":1}
I20260806 15:53:59.942471 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:53:59.932196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:57985 (local address 127.13.223.62:39443)
0806 15:53:59.932353 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:53:59.932358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:53:59.932377 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:53:59.932412 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:53:59.932415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:53:59.932459 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:53:59.932632 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:53:59.935228 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.935749 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.937203 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:53:59.937404 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:53:59.937484 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:53:59.939027 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:53:59.939030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:53:59.939032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:53:59.939055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:53:59.940590 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.941127 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.941130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.941132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.941210 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:53:59.941486 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:53:59.941488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:53:59.941490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:53:59.941748 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:53:59.941831 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 15:53:59.942085 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 15:53:59.942239 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:53:59.959575 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39443
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:00.079420 16632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:00.079748 16632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:00.079829 16632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:00.083952 16632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:00.084059 16632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:00.084138 16632 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:00.084223 16632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:00.089890 16632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:00.091027 16632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:00.092104 16632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:00.102267 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:00.102267 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:00.102490 16632 server_base.cc:1034] running on GCE node
W20260806 15:54:00.102299 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:00.103447 16632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:00.104179 16632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:00.105382 16632 hybrid_clock.cc:648] HybridClock initialized: now 1786031640105337 us; error 58 us; skew 500 ppm
Aug 06 15:54:00 dist-test-slave-rwrg krb5kdc[15975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:00.108457 16632 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:00.109510 16632 webserver.cc:460] Webserver started at http://127.13.223.3:41975/ using document root <none> and password file <none>
I20260806 15:54:00.110112 16632 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:00.110178 16632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:00.110347 16632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:00.112443 16632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "aaf4398cf7c4476c8c1829b347e11366"
format_stamp: "Formatted at 2026-08-06 15:54:00 on dist-test-slave-rwrg"
server_key: "2986ac988843bdd536ef9b1a1eb8d507"
server_key_iv: "082d2eac3cb3e61721e53d6ac9cb7a48"
server_key_version: "encryptionkey@0"
I20260806 15:54:00.113133 16632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "aaf4398cf7c4476c8c1829b347e11366"
format_stamp: "Formatted at 2026-08-06 15:54:00 on dist-test-slave-rwrg"
server_key: "2986ac988843bdd536ef9b1a1eb8d507"
server_key_iv: "082d2eac3cb3e61721e53d6ac9cb7a48"
server_key_version: "encryptionkey@0"
I20260806 15:54:00.116983 16632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:00.119468 16654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:00.120857 16632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:00.120994 16632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "aaf4398cf7c4476c8c1829b347e11366"
format_stamp: "Formatted at 2026-08-06 15:54:00 on dist-test-slave-rwrg"
server_key: "2986ac988843bdd536ef9b1a1eb8d507"
server_key_iv: "082d2eac3cb3e61721e53d6ac9cb7a48"
server_key_version: "encryptionkey@0"
I20260806 15:54:00.121114 16632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:00.135129 16632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:00.138244 16632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:00.138461 16632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:00.139999 16632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:00.140074 16632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:00.140144 16632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:00.140200 16632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:54:00 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:00.152083 16632 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:41593
I20260806 15:54:00.152158 16840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:41593 every 8 connection(s)
I20260806 15:54:00.153352 16632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:00.156366 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.140858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:58829 (local address 127.13.223.62:39443)
0806 15:54:00.141014 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.141018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.142815 (+ 1797us) server_negotiation.cc:407] Connection header received
0806 15:54:00.144189 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.144192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.144256 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.144354 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:00.146230 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.146881 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.148825 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.149040 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.149138 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.152370 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:00.152375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:00.152378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:00.152406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:00.154215 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.154739 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.154743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.154746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.154838 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.155233 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.155237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.155239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.155440 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:00.155551 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.156057 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.156195 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:00.157217 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.141931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.13.223.3:58829)
0806 15:54:00.142630 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.142694 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.143933 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.144596 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.144606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.145177 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:00.146022 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.146049 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.147021 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.147028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.148695 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.148707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.149209 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.149215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.149447 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.150247 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:00.150276 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:00.152190 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:00.154323 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:00.154329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:00.154344 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:00.154618 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:00.154944 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:00.154949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:00.154952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:00.155109 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:00.155563 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:00.155571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:00.155823 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.156016 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.156337 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":144,"threads_started":1}
I20260806 15:54:00.157531 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16632
I20260806 15:54:00.157666 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:00.157976 14204 external_mini_cluster.cc:1428] Setting key 03ac86b2a26997ff1cc5b1303492ff2d
I20260806 15:54:00.158870 16841 heartbeater.cc:347] Connected to a master server at 127.13.223.62:39443
I20260806 15:54:00.159066 16841 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:00.159682 16841 heartbeater.cc:511] Master 127.13.223.62:39443 requested a full tablet report, sending...
I20260806 15:54:00.160837 16049 ts_manager.cc:194] Registered new tserver with Master: aaf4398cf7c4476c8c1829b347e11366 (127.13.223.3:41593)
I20260806 15:54:00.161542 16049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:58829
I20260806 15:54:00.168295 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.161121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:39277 (local address 127.13.223.62:39443)
0806 15:54:00.161242 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.161247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.161423 (+ 176us) server_negotiation.cc:407] Connection header received
0806 15:54:00.161541 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.161545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.161584 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.161675 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:00.162179 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.162690 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.163508 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.163741 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.163857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.165061 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:00.165065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:00.165068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:00.165101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:00.166618 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.167038 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.167041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.167043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.167130 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.167437 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.167440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.167442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.167638 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:00.167753 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.168014 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.168135 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 15:54:00.168305 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.161060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.13.223.3:39277)
0806 15:54:00.161343 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.161355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.161465 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.161787 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.161789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.161964 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:00.162061 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.162067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.162815 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.162818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.163387 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.163394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.163871 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.163875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.163972 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.164472 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:00.164483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:00.164924 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:00.166725 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:00.166728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:00.166731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:00.166925 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:00.167228 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:00.167231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:00.167233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:00.167322 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:00.167745 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:00.167749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:00.167849 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.167957 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.168104 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
I20260806 15:54:00.171998 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 15:54:00 dist-test-slave-rwrg krb5kdc[15975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 15:54:00 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:00.199928 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.191338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.13.223.62:39443)
0806 15:54:00.191474 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.191478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.191489 (+ 11us) server_negotiation.cc:407] Connection header received
0806 15:54:00.191531 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.191534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.191573 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.191640 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:00.192339 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.192933 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.193754 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.193958 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.194038 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.196873 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:00.196876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:00.196878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:00.196902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:00.198251 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.198682 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.198686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.198688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.198774 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.199053 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.199057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.199059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.199250 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:00.199396 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.199639 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.199755 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:54:00.203027 16049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:00.206591 16049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:00.222371 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.215493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:41593 (local address 127.0.0.1:33648)
0806 15:54:00.216449 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.216462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.216637 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.217555 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.217562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.217585 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:00.217686 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.217703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.218892 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.218896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.220664 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.220676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.221811 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.221815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.221920 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.221947 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.222039 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.222193 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":106,"thread_start_us":121,"threads_started":1}
I20260806 15:54:00.222476 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.216263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.13.223.2:44575)
0806 15:54:00.217210 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.217222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.217244 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:54:00.217316 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.217320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.217500 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.217666 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:00.217969 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.219367 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.221006 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.221624 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.221763 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.221945 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.222198 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.222311 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":137,"threads_started":1}
I20260806 15:54:00.222718 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.215524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.13.223.1:34413)
0806 15:54:00.216733 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.216742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.216769 (+ 27us) server_negotiation.cc:407] Connection header received
0806 15:54:00.216849 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.216855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.217079 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.217240 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:00.217845 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.219042 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.221027 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.221809 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.221901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.222074 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.222459 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.222576 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":107,"threads_started":1}
I20260806 15:54:00.222903 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.215784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44575 (local address 127.0.0.1:47754)
0806 15:54:00.216853 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.216866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.216938 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.217665 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.217668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.217688 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:00.217836 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.217844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.219483 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.219486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.220883 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.220891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.221763 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.221767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.221891 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.221945 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.222068 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.222763 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":602,"thread_start_us":101,"threads_started":1}
I20260806 15:54:00.222939 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.215408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:34413 (local address 127.0.0.1:37240)
0806 15:54:00.216240 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.216289 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.216416 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.217246 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.217252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.217276 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:00.217418 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.217426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.219165 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.219169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.220883 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.220894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.222138 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.222144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.222227 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.222247 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.222315 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.222370 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":79,"thread_start_us":81,"threads_started":1}
I20260806 15:54:00.224931 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.216699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.13.223.3:41593)
0806 15:54:00.217022 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.217035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.217062 (+ 27us) server_negotiation.cc:407] Connection header received
0806 15:54:00.217161 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.217191 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.217388 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.217565 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:00.217821 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.218742 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.220805 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.221659 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.224437 (+ 2778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.224633 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.224683 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.224796 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":130,"threads_started":1}
I20260806 15:54:00.225983 16506 tablet_service.cc:1467] Processing CreateTablet for tablet aa988e0ef9c9475db2baa3628251dfb5 (DEFAULT_TABLE table=test-table [id=03f8eb82edfb4eecb5d8996cc09eb989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:00.226507 16260 tablet_service.cc:1467] Processing CreateTablet for tablet aa988e0ef9c9475db2baa3628251dfb5 (DEFAULT_TABLE table=test-table [id=03f8eb82edfb4eecb5d8996cc09eb989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:00.227116 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa988e0ef9c9475db2baa3628251dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.227576 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa988e0ef9c9475db2baa3628251dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.227669 16738 tablet_service.cc:1467] Processing CreateTablet for tablet aa988e0ef9c9475db2baa3628251dfb5 (DEFAULT_TABLE table=test-table [id=03f8eb82edfb4eecb5d8996cc09eb989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:00.228771 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa988e0ef9c9475db2baa3628251dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.234400 16880 tablet_bootstrap.cc:492] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Bootstrap starting.
I20260806 15:54:00.236022 16881 tablet_bootstrap.cc:492] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Bootstrap starting.
I20260806 15:54:00.236686 16880 tablet_bootstrap.cc:654] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.236723 16883 tablet_bootstrap.cc:492] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Bootstrap starting.
I20260806 15:54:00.238284 16881 tablet_bootstrap.cc:654] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.238271 16880 log.cc:824] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:00.238754 16883 tablet_bootstrap.cc:654] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.239195 16881 log.cc:824] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:00.239563 16883 log.cc:824] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:00.240353 16880 tablet_bootstrap.cc:492] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: No bootstrap required, opened a new log
I20260806 15:54:00.240602 16880 ts_tablet_manager.cc:1397] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260806 15:54:00.241137 16881 tablet_bootstrap.cc:492] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: No bootstrap required, opened a new log
I20260806 15:54:00.241358 16881 ts_tablet_manager.cc:1397] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 15:54:00.242085 16883 tablet_bootstrap.cc:492] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: No bootstrap required, opened a new log
I20260806 15:54:00.242267 16883 ts_tablet_manager.cc:1397] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260806 15:54:00.244695 16880 raft_consensus.cc:348] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.245080 16880 raft_consensus.cc:374] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.245152 16880 raft_consensus.cc:729] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c245300614bde9658284dd52ef798, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.245602 16880 consensus_queue.cc:260] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.245661 16881 raft_consensus.cc:348] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.245951 16881 raft_consensus.cc:374] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.246017 16881 raft_consensus.cc:729] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6e1c4abf8648c89d726fdd7bf7ab85, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.246011 16883 raft_consensus.cc:348] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.246297 16883 raft_consensus.cc:374] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.246361 16883 raft_consensus.cc:729] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4398cf7c4476c8c1829b347e11366, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.246479 16881 consensus_queue.cc:260] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.246666 16880 ts_tablet_manager.cc:1428] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:54:00.246812 16883 consensus_queue.cc:260] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.247068 16618 heartbeater.cc:503] Master 127.13.223.62:39443 was elected leader, sending a full tablet report...
I20260806 15:54:00.247243 16376 heartbeater.cc:503] Master 127.13.223.62:39443 was elected leader, sending a full tablet report...
I20260806 15:54:00.247521 16841 heartbeater.cc:503] Master 127.13.223.62:39443 was elected leader, sending a full tablet report...
I20260806 15:54:00.247822 16883 ts_tablet_manager.cc:1428] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 15:54:00.248828 16881 ts_tablet_manager.cc:1428] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 15:54:00.400398 16895 raft_consensus.cc:482] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:00.400619 16895 raft_consensus.cc:504] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.401803 16895 leader_election.cc:288] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413), 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
W20260806 15:54:00.404542 16842 tablet.cc:2369] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:00.406164 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.402152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:34413 (local address 127.13.223.3:55161)
0806 15:54:00.402363 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.402380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.402484 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.402787 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.402789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.402807 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:00.402930 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.402936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.403819 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.403822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.404920 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.404930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.405702 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.405706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.405809 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.405830 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.405945 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.406018 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 15:54:00.406164 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.402298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:55161 (local address 127.13.223.1:34413)
0806 15:54:00.402433 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.402444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.402458 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:00.402638 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.402641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.402686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.402774 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:00.403042 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.403681 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.405048 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.405560 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.405716 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.405772 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.405961 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.406034 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:54:00.406967 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa988e0ef9c9475db2baa3628251dfb5" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" is_pre_election: true
I20260806 15:54:00.407321 16300 raft_consensus.cc:2454] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 0.
I20260806 15:54:00.407470 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.402234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:57601 (local address 127.13.223.2:44575)
0806 15:54:00.402374 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.402379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.403119 (+ 740us) server_negotiation.cc:407] Connection header received
0806 15:54:00.403309 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.403312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.403363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.403450 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:00.404302 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.405075 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.406420 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.406933 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.407026 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.407065 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.407285 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.407349 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:54:00.407491 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.402160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44575 (local address 127.13.223.3:57601)
0806 15:54:00.403012 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.403027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.403157 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.403471 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.403474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.403490 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:00.404140 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.404149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.405210 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.405213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.406229 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.406242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.407062 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.407066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.407169 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.407193 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.407272 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.407335 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":335,"thread_start_us":209,"threads_started":1}
I20260806 15:54:00.407884 16659 leader_election.cc:302] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d6e1c4abf8648c89d726fdd7bf7ab85, aaf4398cf7c4476c8c1829b347e11366; no voters:
I20260806 15:54:00.408146 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa988e0ef9c9475db2baa3628251dfb5" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c245300614bde9658284dd52ef798" is_pre_election: true
I20260806 15:54:00.408177 16895 raft_consensus.cc:2781] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:00.408283 16895 raft_consensus.cc:482] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:00.408353 16895 raft_consensus.cc:3037] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.408449 16541 raft_consensus.cc:2454] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 0.
I20260806 15:54:00.409498 16895 raft_consensus.cc:504] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.409900 16895 leader_election.cc:288] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 election: Requested vote from peers 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413), 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
I20260806 15:54:00.410210 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa988e0ef9c9475db2baa3628251dfb5" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85"
I20260806 15:54:00.410305 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa988e0ef9c9475db2baa3628251dfb5" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c245300614bde9658284dd52ef798"
I20260806 15:54:00.410346 16300 raft_consensus.cc:3037] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.410456 16541 raft_consensus.cc:3037] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.411479 16300 raft_consensus.cc:2454] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 1.
I20260806 15:54:00.411486 16541 raft_consensus.cc:2454] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 1.
I20260806 15:54:00.411790 16659 leader_election.cc:302] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d6e1c4abf8648c89d726fdd7bf7ab85, aaf4398cf7c4476c8c1829b347e11366; no voters:
I20260806 15:54:00.412012 16895 raft_consensus.cc:2781] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:00.412245 16895 raft_consensus.cc:686] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 1 LEADER]: Becoming Leader. State: Replica: aaf4398cf7c4476c8c1829b347e11366, State: Running, Role: LEADER
I20260806 15:54:00.412638 16895 consensus_queue.cc:237] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } }
I20260806 15:54:00.415791 16049 catalog_manager.cc:5495] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf4398cf7c4476c8c1829b347e11366 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } health_report { overall_health: UNKNOWN } } }
W20260806 15:54:00.427469 16622 tablet.cc:2369] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:00.450673 16378 tablet.cc:2369] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:00.465495 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.461078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.13.223.3:41593)
0806 15:54:00.461238 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.461241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.461255 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:00.461494 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.461497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.461601 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.461729 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:00.462038 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.462565 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.463490 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.463756 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.463990 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.464397 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:00.464946 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:00.465097 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.465254 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.465321 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 15:54:00.471993 16300 raft_consensus.cc:1261] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Refusing update from remote peer aaf4398cf7c4476c8c1829b347e11366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:00.472783 16899 consensus_queue.cc:1035] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:00.473714 16541 raft_consensus.cc:1261] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Refusing update from remote peer aaf4398cf7c4476c8c1829b347e11366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:00.474481 16899 consensus_queue.cc:1035] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:00.483635 16908 mvcc.cc:205] Tried to move back new op lower bound from 7315585599365087232 to 7315585599132667904. Current Snapshot: MvccSnapshot[applied={T|T < 7315585599365087232 or (T in {7315585599365087232})}]
I20260806 15:54:00.483860 16906 mvcc.cc:205] Tried to move back new op lower bound from 7315585599365087232 to 7315585599132667904. Current Snapshot: MvccSnapshot[applied={T|T < 7315585599365087232 or (T in {7315585599365087232})}]
I20260806 15:54:00.493327 16044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:00.493585 16044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:00.496218 16044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: Sending DeleteTablet for 3 replicas of tablet aa988e0ef9c9475db2baa3628251dfb5
I20260806 15:54:00.497131 16260 tablet_service.cc:1514] Processing DeleteTablet for tablet aa988e0ef9c9475db2baa3628251dfb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:37240
I20260806 15:54:00.497203 16738 tablet_service.cc:1514] Processing DeleteTablet for tablet aa988e0ef9c9475db2baa3628251dfb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33648
I20260806 15:54:00.497876 16920 tablet_replica.cc:330] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: stopping tablet replica
I20260806 15:54:00.498281 16921 tablet_replica.cc:330] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: stopping tablet replica
I20260806 15:54:00.498281 16920 raft_consensus.cc:2229] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:00.498232 16506 tablet_service.cc:1514] Processing DeleteTablet for tablet aa988e0ef9c9475db2baa3628251dfb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:47754
I20260806 15:54:00.498034 16044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 15:54:00.498833 16921 raft_consensus.cc:2229] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:00.499073 16920 raft_consensus.cc:2258] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:00.499181 16921 raft_consensus.cc:2258] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 15:54:00.499295 16044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:00.500897 16921 ts_tablet_manager.cc:1905] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:00.503278 16920 ts_tablet_manager.cc:1905] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:00.504163 16921 ts_tablet_manager.cc:1918] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:00.504304 16921 log.cc:1196] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/aa988e0ef9c9475db2baa3628251dfb5
I20260806 15:54:00.504763 16921 ts_tablet_manager.cc:1939] T aa988e0ef9c9475db2baa3628251dfb5 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Deleting consensus metadata
I20260806 15:54:00.505270 16738 tablet_service.cc:1467] Processing CreateTablet for tablet 0f682fb679c34f0489ab090dccdbafc3 (DEFAULT_TABLE table=test-table [id=55cd9747161341adba0ca4a3601eb7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:00.505524 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f682fb679c34f0489ab090dccdbafc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.505563 16260 tablet_service.cc:1467] Processing CreateTablet for tablet 0f682fb679c34f0489ab090dccdbafc3 (DEFAULT_TABLE table=test-table [id=55cd9747161341adba0ca4a3601eb7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:00.505789 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f682fb679c34f0489ab090dccdbafc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.506065 16027 catalog_manager.cc:4841] TS 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413): tablet aa988e0ef9c9475db2baa3628251dfb5 (table test-table [id=03f8eb82edfb4eecb5d8996cc09eb989]) successfully deleted
I20260806 15:54:00.507648 16881 tablet_bootstrap.cc:492] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Bootstrap starting.
I20260806 15:54:00.507767 16883 tablet_bootstrap.cc:492] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Bootstrap starting.
I20260806 15:54:00.508706 16883 tablet_bootstrap.cc:654] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.508705 16881 tablet_bootstrap.cc:654] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.509541 16920 ts_tablet_manager.cc:1918] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:00.509630 16920 log.cc:1196] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/aa988e0ef9c9475db2baa3628251dfb5
I20260806 15:54:00.509967 16920 ts_tablet_manager.cc:1939] T aa988e0ef9c9475db2baa3628251dfb5 P aaf4398cf7c4476c8c1829b347e11366: Deleting consensus metadata
I20260806 15:54:00.510133 16506 tablet_service.cc:1467] Processing CreateTablet for tablet 0f682fb679c34f0489ab090dccdbafc3 (DEFAULT_TABLE table=test-table [id=55cd9747161341adba0ca4a3601eb7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:00.510455 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f682fb679c34f0489ab090dccdbafc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.510670 16883 tablet_bootstrap.cc:492] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: No bootstrap required, opened a new log
I20260806 15:54:00.510763 16883 ts_tablet_manager.cc:1397] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:00.511296 16032 catalog_manager.cc:4841] TS aaf4398cf7c4476c8c1829b347e11366 (127.13.223.3:41593): tablet aa988e0ef9c9475db2baa3628251dfb5 (table test-table [id=03f8eb82edfb4eecb5d8996cc09eb989]) successfully deleted
I20260806 15:54:00.511539 16883 raft_consensus.cc:348] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.511658 16883 raft_consensus.cc:374] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.511691 16883 raft_consensus.cc:729] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4398cf7c4476c8c1829b347e11366, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.511881 16883 consensus_queue.cc:260] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.512259 16883 ts_tablet_manager.cc:1428] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:54:00.513383 16881 tablet_bootstrap.cc:492] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: No bootstrap required, opened a new log
I20260806 15:54:00.513471 16881 ts_tablet_manager.cc:1397] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260806 15:54:00.514627 16922 tablet_replica.cc:330] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: stopping tablet replica
I20260806 15:54:00.514784 16881 raft_consensus.cc:348] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.514894 16881 raft_consensus.cc:374] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.514920 16881 raft_consensus.cc:729] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6e1c4abf8648c89d726fdd7bf7ab85, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.515036 16922 raft_consensus.cc:2229] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:00.515070 16881 consensus_queue.cc:260] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.515309 16881 ts_tablet_manager.cc:1428] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:00.515412 16922 raft_consensus.cc:2258] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:00.515563 16880 tablet_bootstrap.cc:492] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Bootstrap starting.
I20260806 15:54:00.517103 16880 tablet_bootstrap.cc:654] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.519162 16922 ts_tablet_manager.cc:1905] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:00.519191 16880 tablet_bootstrap.cc:492] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: No bootstrap required, opened a new log
I20260806 15:54:00.519311 16880 ts_tablet_manager.cc:1397] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260806 15:54:00.519861 16880 raft_consensus.cc:348] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.519984 16880 raft_consensus.cc:374] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.520028 16880 raft_consensus.cc:729] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c245300614bde9658284dd52ef798, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.520174 16880 consensus_queue.cc:260] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.520437 16880 ts_tablet_manager.cc:1428] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:54:00.523866 16922 ts_tablet_manager.cc:1918] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:00.523983 16922 log.cc:1196] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/aa988e0ef9c9475db2baa3628251dfb5
I20260806 15:54:00.524315 16922 ts_tablet_manager.cc:1939] T aa988e0ef9c9475db2baa3628251dfb5 P 003c245300614bde9658284dd52ef798: Deleting consensus metadata
I20260806 15:54:00.525636 16030 catalog_manager.cc:4841] TS 003c245300614bde9658284dd52ef798 (127.13.223.2:44575): tablet aa988e0ef9c9475db2baa3628251dfb5 (table test-table [id=03f8eb82edfb4eecb5d8996cc09eb989]) successfully deleted
I20260806 15:54:00.603878 16895 raft_consensus.cc:482] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:00.604068 16895 raft_consensus.cc:504] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.604698 16895 leader_election.cc:288] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413), 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
I20260806 15:54:00.605238 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f682fb679c34f0489ab090dccdbafc3" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c245300614bde9658284dd52ef798" is_pre_election: true
I20260806 15:54:00.605300 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f682fb679c34f0489ab090dccdbafc3" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" is_pre_election: true
I20260806 15:54:00.605486 16541 raft_consensus.cc:2454] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 0.
I20260806 15:54:00.605522 16300 raft_consensus.cc:2454] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 0.
I20260806 15:54:00.605918 16659 leader_election.cc:302] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d6e1c4abf8648c89d726fdd7bf7ab85, aaf4398cf7c4476c8c1829b347e11366; no voters:
I20260806 15:54:00.606117 16895 raft_consensus.cc:2781] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:00.606204 16895 raft_consensus.cc:482] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:00.606277 16895 raft_consensus.cc:3037] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.607693 16895 raft_consensus.cc:504] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.608124 16895 leader_election.cc:288] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 election: Requested vote from peers 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413), 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
I20260806 15:54:00.608528 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f682fb679c34f0489ab090dccdbafc3" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85"
I20260806 15:54:00.608680 16300 raft_consensus.cc:3037] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.608611 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f682fb679c34f0489ab090dccdbafc3" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c245300614bde9658284dd52ef798"
I20260806 15:54:00.608771 16541 raft_consensus.cc:3037] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.610082 16541 raft_consensus.cc:2454] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 1.
I20260806 15:54:00.610363 16300 raft_consensus.cc:2454] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 1.
I20260806 15:54:00.610406 16661 leader_election.cc:302] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003c245300614bde9658284dd52ef798, aaf4398cf7c4476c8c1829b347e11366; no voters:
I20260806 15:54:00.610670 16895 raft_consensus.cc:2781] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:00.610774 16895 raft_consensus.cc:686] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 1 LEADER]: Becoming Leader. State: Replica: aaf4398cf7c4476c8c1829b347e11366, State: Running, Role: LEADER
I20260806 15:54:00.610926 16895 consensus_queue.cc:237] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.613103 16049 catalog_manager.cc:5495] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf4398cf7c4476c8c1829b347e11366 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } health_report { overall_health: HEALTHY } } }
Aug 06 15:54:00 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:00.673942 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.662628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39443 (local address 127.0.0.1:45888)
0806 15:54:00.663155 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.663169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.663284 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.663610 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.663612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.664106 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:00.664208 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.664214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.665540 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.665543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.666234 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.666244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.666969 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.666972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.667055 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.667788 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:00.667800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:00.669665 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:00.672138 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:00.672147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:00.672152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:00.672406 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:00.672826 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:00.672829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:00.672830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:00.672917 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:00.673445 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:00.673451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:00.673564 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.673666 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.673791 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":2,"thread_start_us":157,"threads_started":1}
I20260806 15:54:00.674232 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.662729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.13.223.62:39443)
0806 15:54:00.662902 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.662908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.663270 (+ 362us) server_negotiation.cc:407] Connection header received
0806 15:54:00.663377 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.663381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.663438 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.663544 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:00.664359 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.665376 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.666352 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.666825 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.666957 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.669825 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:00.669831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:00.669834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:00.669877 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:00.672017 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.672576 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.672580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.672582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.672697 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.673042 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.673046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.673048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.673322 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:00.673445 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.673783 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.673948 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 15:54:00.679553 16049 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:45888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 15:54:00.680337 16049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:00.686415 16260 tablet_service.cc:1467] Processing CreateTablet for tablet da7bdbcf748c41489600d64f30261086 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c46be09993c84072811348b69645ddbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:00.686828 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bdbcf748c41489600d64f30261086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.686805 16738 tablet_service.cc:1467] Processing CreateTablet for tablet da7bdbcf748c41489600d64f30261086 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c46be09993c84072811348b69645ddbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:00.687493 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bdbcf748c41489600d64f30261086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.689790 16881 tablet_bootstrap.cc:492] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Bootstrap starting.
I20260806 15:54:00.690416 16506 tablet_service.cc:1467] Processing CreateTablet for tablet da7bdbcf748c41489600d64f30261086 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c46be09993c84072811348b69645ddbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:00.691088 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bdbcf748c41489600d64f30261086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:00.691870 16881 tablet_bootstrap.cc:654] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.692751 16883 tablet_bootstrap.cc:492] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: Bootstrap starting.
I20260806 15:54:00.693783 16883 tablet_bootstrap.cc:654] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.694121 16881 tablet_bootstrap.cc:492] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: No bootstrap required, opened a new log
I20260806 15:54:00.694202 16881 ts_tablet_manager.cc:1397] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260806 15:54:00.694860 16881 raft_consensus.cc:348] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.694990 16881 raft_consensus.cc:374] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.694973 16880 tablet_bootstrap.cc:492] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: Bootstrap starting.
I20260806 15:54:00.695029 16881 raft_consensus.cc:729] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6e1c4abf8648c89d726fdd7bf7ab85, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.695169 16881 consensus_queue.cc:260] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.695444 16881 ts_tablet_manager.cc:1428] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:54:00.696627 16880 tablet_bootstrap.cc:654] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:00.698130 16894 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.698338 16894 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.699833 16883 tablet_bootstrap.cc:492] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: No bootstrap required, opened a new log
I20260806 15:54:00.699924 16883 ts_tablet_manager.cc:1397] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 15:54:00.700568 16883 raft_consensus.cc:348] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.700692 16883 raft_consensus.cc:374] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.700726 16883 raft_consensus.cc:729] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4398cf7c4476c8c1829b347e11366, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.700889 16883 consensus_queue.cc:260] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.701346 16880 tablet_bootstrap.cc:492] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: No bootstrap required, opened a new log
I20260806 15:54:00.701433 16880 ts_tablet_manager.cc:1397] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260806 15:54:00.701428 16895 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.701550 16895 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.702024 16880 raft_consensus.cc:348] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.702132 16880 raft_consensus.cc:374] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:00.702159 16880 raft_consensus.cc:729] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c245300614bde9658284dd52ef798, State: Initialized, Role: FOLLOWER
I20260806 15:54:00.702272 16880 consensus_queue.cc:260] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.702534 16893 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.702723 16893 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.702986 16880 ts_tablet_manager.cc:1428] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 15:54:00.703972 16883 ts_tablet_manager.cc:1428] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 15:54:00.879874 16895 raft_consensus.cc:482] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:00.880016 16895 raft_consensus.cc:504] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.880573 16895 leader_election.cc:288] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413), 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
I20260806 15:54:00.881080 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bdbcf748c41489600d64f30261086" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c245300614bde9658284dd52ef798" is_pre_election: true
I20260806 15:54:00.881078 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bdbcf748c41489600d64f30261086" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" is_pre_election: true
I20260806 15:54:00.881237 16541 raft_consensus.cc:2454] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 0.
I20260806 15:54:00.881237 16300 raft_consensus.cc:2454] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 0.
I20260806 15:54:00.881534 16661 leader_election.cc:302] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003c245300614bde9658284dd52ef798, aaf4398cf7c4476c8c1829b347e11366; no voters:
I20260806 15:54:00.881795 16895 raft_consensus.cc:2781] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:00.881852 16895 raft_consensus.cc:482] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:00.881878 16895 raft_consensus.cc:3037] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.883154 16895 raft_consensus.cc:504] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.883525 16895 leader_election.cc:288] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 election: Requested vote from peers 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413), 003c245300614bde9658284dd52ef798 (127.13.223.2:44575)
I20260806 15:54:00.883900 16300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bdbcf748c41489600d64f30261086" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85"
I20260806 15:54:00.883957 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bdbcf748c41489600d64f30261086" candidate_uuid: "aaf4398cf7c4476c8c1829b347e11366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c245300614bde9658284dd52ef798"
I20260806 15:54:00.884029 16300 raft_consensus.cc:3037] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.884145 16541 raft_consensus.cc:3037] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:00.885010 16300 raft_consensus.cc:2454] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 1.
I20260806 15:54:00.885227 16541 raft_consensus.cc:2454] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4398cf7c4476c8c1829b347e11366 in term 1.
I20260806 15:54:00.885318 16659 leader_election.cc:302] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d6e1c4abf8648c89d726fdd7bf7ab85, aaf4398cf7c4476c8c1829b347e11366; no voters:
I20260806 15:54:00.885514 16895 raft_consensus.cc:2781] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:00.885624 16895 raft_consensus.cc:686] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [term 1 LEADER]: Becoming Leader. State: Replica: aaf4398cf7c4476c8c1829b347e11366, State: Running, Role: LEADER
I20260806 15:54:00.885757 16895 consensus_queue.cc:237] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } }
I20260806 15:54:00.886401 16899 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: TxnStatusTablet state changed. Reason: New leader aaf4398cf7c4476c8c1829b347e11366. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.886503 16899 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:00.886787 16941 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 15:54:00.886960 16941 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 15:54:00.887504 16049 catalog_manager.cc:5495] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf4398cf7c4476c8c1829b347e11366 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } health_report { overall_health: HEALTHY } } }
I20260806 15:54:00.900966 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.897265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:41593 (local address 127.0.0.1:33678)
0806 15:54:00.897431 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.897449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.897556 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.897848 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.897851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.897860 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:00.897963 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.897968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.898975 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.898978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.899602 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.899611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.900163 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.900166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.900229 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.900272 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:00.900638 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:00.900652 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.900736 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.900829 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 15:54:00.901335 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.897403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.13.223.3:41593)
0806 15:54:00.897539 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.897552 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.897568 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:00.897666 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.897669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.897721 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.897802 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:00.898067 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.898852 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.899713 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.900054 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.900157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.900397 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:00.900477 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:00.900610 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.901160 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.901215 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:54:00.903584 16300 raft_consensus.cc:1261] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Refusing update from remote peer aaf4398cf7c4476c8c1829b347e11366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:00.903627 16541 raft_consensus.cc:1261] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Refusing update from remote peer aaf4398cf7c4476c8c1829b347e11366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:00.904009 16899 consensus_queue.cc:1035] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:00.904166 16895 consensus_queue.cc:1035] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:00.905704 16894 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: TxnStatusTablet state changed. Reason: New leader aaf4398cf7c4476c8c1829b347e11366. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.905819 16894 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.907348 16893 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: TxnStatusTablet state changed. Reason: New leader aaf4398cf7c4476c8c1829b347e11366. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.907461 16893 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.908701 16895 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.908806 16895 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:00.909293 16902 mvcc.cc:205] Tried to move back new op lower bound from 7315585601137205248 to 7315585601069363200. Current Snapshot: MvccSnapshot[applied={T|T < 7315585601137205248 or (T in {7315585601137205248})}]
I20260806 15:54:00.909453 16893 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.909552 16893 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P 003c245300614bde9658284dd52ef798: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.910606 16895 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.910708 16895 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P aaf4398cf7c4476c8c1829b347e11366: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:00.911641 16894 tablet_replica.cc:439] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf4398cf7c4476c8c1829b347e11366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 } } peers { permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 } } peers { permanent_uuid: "aaf4398cf7c4476c8c1829b347e11366" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 41593 } } }
I20260806 15:54:00.911743 16894 tablet_replica.cc:442] T da7bdbcf748c41489600d64f30261086 P 2d6e1c4abf8648c89d726fdd7bf7ab85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:00.925634 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.921692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:36545 (local address 127.13.223.3:41593)
0806 15:54:00.921823 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.921829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.922165 (+ 336us) server_negotiation.cc:407] Connection header received
0806 15:54:00.922316 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.922319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.922364 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.922441 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:00.922658 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.923144 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.924319 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.924684 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.924787 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.925083 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:00.925180 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:00.925262 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.925419 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.925483 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260806 15:54:00.925684 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.921615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:41593 (local address 127.13.223.3:36545)
0806 15:54:00.922069 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:00.922081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:00.922222 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:00.922455 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:00.922458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:00.922465 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:00.922560 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.922565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.923304 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.923307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.924184 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:00.924196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.924804 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.924808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:00.924894 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.924943 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:00.925288 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:00.925296 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 15:54:00.925390 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 15:54:00.925474 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":1,"thread_start_us":174,"threads_started":1}
I20260806 15:54:00.935238 16541 raft_consensus.cc:1261] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Refusing update from remote peer aaf4398cf7c4476c8c1829b347e11366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:00.935637 16947 consensus_queue.cc:1035] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003c245300614bde9658284dd52ef798" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:00.935694 16300 raft_consensus.cc:1261] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Refusing update from remote peer aaf4398cf7c4476c8c1829b347e11366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:00.936043 16947 consensus_queue.cc:1035] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6e1c4abf8648c89d726fdd7bf7ab85" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:00.975807 16049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:00.975979 16049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:00.978044 16049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8ef6559ad6c485b9be186644e032dd3: Sending DeleteTablet for 3 replicas of tablet 0f682fb679c34f0489ab090dccdbafc3
I20260806 15:54:00.978562 16260 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f682fb679c34f0489ab090dccdbafc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:37240
I20260806 15:54:00.978655 16506 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f682fb679c34f0489ab090dccdbafc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:47754
I20260806 15:54:00.978753 16921 tablet_replica.cc:330] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: stopping tablet replica
I20260806 15:54:00.978833 16922 tablet_replica.cc:330] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: stopping tablet replica
I20260806 15:54:00.978875 16921 raft_consensus.cc:2229] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:00.978955 16922 raft_consensus.cc:2229] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:00.978991 16921 raft_consensus.cc:2258] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:00.979012 16737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f682fb679c34f0489ab090dccdbafc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33648
I20260806 15:54:00.979066 16922 raft_consensus.cc:2258] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:00.979171 16920 tablet_replica.cc:330] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: stopping tablet replica
I20260806 15:54:00.979293 16920 raft_consensus.cc:2229] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:00.979483 16920 raft_consensus.cc:2258] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:00.979913 16922 ts_tablet_manager.cc:1905] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:00.980665 16920 ts_tablet_manager.cc:1905] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:00.982774 16922 ts_tablet_manager.cc:1918] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:00.982872 16922 log.cc:1196] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/0f682fb679c34f0489ab090dccdbafc3
I20260806 15:54:00.983135 16922 ts_tablet_manager.cc:1939] T 0f682fb679c34f0489ab090dccdbafc3 P 003c245300614bde9658284dd52ef798: Deleting consensus metadata
I20260806 15:54:00.983309 16920 ts_tablet_manager.cc:1918] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:00.983390 16920 log.cc:1196] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/0f682fb679c34f0489ab090dccdbafc3
I20260806 15:54:00.983660 16920 ts_tablet_manager.cc:1939] T 0f682fb679c34f0489ab090dccdbafc3 P aaf4398cf7c4476c8c1829b347e11366: Deleting consensus metadata
I20260806 15:54:00.983891 16030 catalog_manager.cc:4841] TS 003c245300614bde9658284dd52ef798 (127.13.223.2:44575): tablet 0f682fb679c34f0489ab090dccdbafc3 (table test-table [id=55cd9747161341adba0ca4a3601eb7ef]) successfully deleted
I20260806 15:54:00.984268 16032 catalog_manager.cc:4841] TS aaf4398cf7c4476c8c1829b347e11366 (127.13.223.3:41593): tablet 0f682fb679c34f0489ab090dccdbafc3 (table test-table [id=55cd9747161341adba0ca4a3601eb7ef]) successfully deleted
I20260806 15:54:00.984910 16921 ts_tablet_manager.cc:1905] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:00.987154 16921 ts_tablet_manager.cc:1918] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:00.987246 16921 log.cc:1196] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/0f682fb679c34f0489ab090dccdbafc3
I20260806 15:54:00.987483 16921 ts_tablet_manager.cc:1939] T 0f682fb679c34f0489ab090dccdbafc3 P 2d6e1c4abf8648c89d726fdd7bf7ab85: Deleting consensus metadata
I20260806 15:54:00.988070 16027 catalog_manager.cc:4841] TS 2d6e1c4abf8648c89d726fdd7bf7ab85 (127.13.223.1:34413): tablet 0f682fb679c34f0489ab090dccdbafc3 (table test-table [id=55cd9747161341adba0ca4a3601eb7ef]) successfully deleted
Aug 06 15:54:00 dist-test-slave-rwrg krb5kdc[15975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:54:01.000306 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:00.980590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.13.223.1:34413)
0806 15:54:00.980877 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:00.980882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:00.988796 (+ 7914us) server_negotiation.cc:407] Connection header received
0806 15:54:00.988920 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:00.988923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:00.988967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:00.989082 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:00.989797 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.990388 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.991186 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:00.991407 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:00.991512 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:00.994526 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:00.994538 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:00.994559 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:00.994591 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:00.997312 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.998056 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.998063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.998065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.998220 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:00.998608 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:00.998611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:00.998613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:00.999011 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:00.999312 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:00.999786 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 15:54:00.999975 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":144,"threads_started":1}
W20260806 15:54:01.001330 16368 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37252
I20260806 15:54:01.024294 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:01.013052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.13.223.62:39443)
0806 15:54:01.013213 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:01.013219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:01.013611 (+ 392us) server_negotiation.cc:407] Connection header received
0806 15:54:01.013857 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:01.013861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:01.013927 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:01.013993 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:01.015065 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:01.015849 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:01.017248 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:01.017631 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:01.017730 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:01.020122 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:01.020130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:01.020137 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:01.020173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:01.022257 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:01.022776 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:01.022779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:01.022781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:01.022948 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:01.023254 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:01.023257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:01.023259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:01.023484 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:01.023626 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:01.023927 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 15:54:01.024061 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 15:54:01.024772 16049 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45898
I20260806 15:54:01.026285 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16159
I20260806 15:54:01.041391 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16393
I20260806 15:54:01.055416 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16632
I20260806 15:54:01.072237 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15996
2026-08-06T15:54:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3280 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:01 dist-test-slave-rwrg krb5kdc[16968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:01 dist-test-slave-rwrg krb5kdc[16968](info): set up 2 sockets
Aug 06 15:54:01 dist-test-slave-rwrg krb5kdc[16968](info): commencing operation
krb5kdc: starting...
W20260806 15:54:02.347942 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:02 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:02.474700 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43745
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:43745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:02.589779 16990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:02.590036 16990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:02.590076 16990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:02.594004 16990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:02.594053 16990 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 15:54:02.594070 16990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:02.594115 16990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:02.594130 16990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:02.594146 16990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:02.599893 16990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:43745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:02.601159 16990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:02.602246 16990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:02.611057 17004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:02.611029 17002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:02.611094 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:02.611371 16990 server_base.cc:1034] running on GCE node
I20260806 15:54:02.611901 16990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:02.613102 16990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:02.614269 16990 hybrid_clock.cc:648] HybridClock initialized: now 1786031642614233 us; error 50 us; skew 500 ppm
Aug 06 15:54:02 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031642, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:02.620143 16990 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:02.621140 16990 webserver.cc:460] Webserver started at http://127.13.223.62:36857/ using document root <none> and password file <none>
I20260806 15:54:02.621737 16990 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:02.621802 16990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:02.622066 16990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:02.624106 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "5a6413ca40e3420abdb3f3da448f36d3"
format_stamp: "Formatted at 2026-08-06 15:54:02 on dist-test-slave-rwrg"
server_key: "f093ced73710bcdcb80920d4715bcf2a"
server_key_iv: "f9c73c3811f858e2f3bf9388fb2dbdcc"
server_key_version: "encryptionkey@0"
I20260806 15:54:02.624709 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "5a6413ca40e3420abdb3f3da448f36d3"
format_stamp: "Formatted at 2026-08-06 15:54:02 on dist-test-slave-rwrg"
server_key: "f093ced73710bcdcb80920d4715bcf2a"
server_key_iv: "f9c73c3811f858e2f3bf9388fb2dbdcc"
server_key_version: "encryptionkey@0"
I20260806 15:54:02.628638 16990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:54:02.631065 17013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:02.632342 16990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:02.632488 16990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "5a6413ca40e3420abdb3f3da448f36d3"
format_stamp: "Formatted at 2026-08-06 15:54:02 on dist-test-slave-rwrg"
server_key: "f093ced73710bcdcb80920d4715bcf2a"
server_key_iv: "f9c73c3811f858e2f3bf9388fb2dbdcc"
server_key_version: "encryptionkey@0"
I20260806 15:54:02.632628 16990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:02.642683 16990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:02.645725 16990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:02.645936 16990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:02.654155 16990 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:43745
I20260806 15:54:02.654170 17110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:43745 every 8 connection(s)
I20260806 15:54:02.655118 16990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:02.657984 17111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:02.661489 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16990
I20260806 15:54:02.661583 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:02.661860 14204 external_mini_cluster.cc:1428] Setting key dab9e4fd1d3a96f692230afe5b71e500
I20260806 15:54:02.664029 17111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Bootstrap starting.
I20260806 15:54:02.666877 17111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:02.667759 17111 log.cc:824] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:02.670176 17111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: No bootstrap required, opened a new log
Aug 06 15:54:02 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:02.674490 17111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6413ca40e3420abdb3f3da448f36d3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43745 } }
I20260806 15:54:02.674786 17111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:02.674853 17111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6413ca40e3420abdb3f3da448f36d3, State: Initialized, Role: FOLLOWER
I20260806 15:54:02.675417 17111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6413ca40e3420abdb3f3da448f36d3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43745 } }
I20260806 15:54:02.675580 17111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:02.675664 17111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:02.675773 17111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:02.676951 17111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6413ca40e3420abdb3f3da448f36d3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43745 } }
I20260806 15:54:02.677295 17111 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6413ca40e3420abdb3f3da448f36d3; no voters:
I20260806 15:54:02.677551 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:02.663310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.13.223.62:43745)
0806 15:54:02.663745 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:02.663765 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:02.663807 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:54:02.664673 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:02.664696 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:02.665108 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:02.665487 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:02.666138 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.667692 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.668828 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.669509 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.669682 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:02.672121 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:02.672128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:02.672145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:02.672178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:02.674221 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:02.674650 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:02.674656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:02.674662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:02.674809 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:02.675151 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:02.675154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:02.675156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:02.675663 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:02.675859 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:02.676209 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 15:54:02.676526 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":109,"threads_started":1}
I20260806 15:54:02.677650 17111 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:02.677780 17118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:02.678046 17118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [term 1 LEADER]: Becoming Leader. State: Replica: 5a6413ca40e3420abdb3f3da448f36d3, State: Running, Role: LEADER
I20260806 15:54:02.678442 17118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6413ca40e3420abdb3f3da448f36d3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43745 } }
I20260806 15:54:02.678905 17111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:02.679610 17118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6413ca40e3420abdb3f3da448f36d3. Latest consensus state: current_term: 1 leader_uuid: "5a6413ca40e3420abdb3f3da448f36d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6413ca40e3420abdb3f3da448f36d3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43745 } } }
I20260806 15:54:02.679737 17118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:02.680158 17119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a6413ca40e3420abdb3f3da448f36d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6413ca40e3420abdb3f3da448f36d3" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43745 } } }
I20260806 15:54:02.680265 17119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:02.681320 17132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:02.684226 17132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:02.689668 17132 catalog_manager.cc:1347] Generated new cluster ID: ff7047d6ee9d483c81960376d339e576
I20260806 15:54:02.689790 17132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:02.705482 17132 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:02.706424 17132 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:02.715765 17132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Generated new TSK 0
I20260806 15:54:02.716431 17132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:02.749373 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43745
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:02.865214 17149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:02.865490 17149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:02.865579 17149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:02.869589 17149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:02.869647 17149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:02.869742 17149 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 15:54:02.869776 17149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:02.869829 17149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:02.875087 17149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:02.876111 17149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:02.877179 17149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:02.887827 17158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:02.887789 17157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:02.887789 17163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:02.888003 17149 server_base.cc:1034] running on GCE node
I20260806 15:54:02.888486 17149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:02.889315 17149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:02.890543 17149 hybrid_clock.cc:648] HybridClock initialized: now 1786031642890491 us; error 62 us; skew 500 ppm
Aug 06 15:54:02 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031642, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:02.893864 17149 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:02.894740 17149 webserver.cc:460] Webserver started at http://127.13.223.1:35649/ using document root <none> and password file <none>
I20260806 15:54:02.895330 17149 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:02.895423 17149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:02.895648 17149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:02.897861 17149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "fe96b06192d14756aff5ae81f2b640d3"
format_stamp: "Formatted at 2026-08-06 15:54:02 on dist-test-slave-rwrg"
server_key: "6c977c38f0f514331495d1a1b241a11a"
server_key_iv: "cc24a79fadddc971b2a241af0538eae8"
server_key_version: "encryptionkey@0"
I20260806 15:54:02.898473 17149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "fe96b06192d14756aff5ae81f2b640d3"
format_stamp: "Formatted at 2026-08-06 15:54:02 on dist-test-slave-rwrg"
server_key: "6c977c38f0f514331495d1a1b241a11a"
server_key_iv: "cc24a79fadddc971b2a241af0538eae8"
server_key_version: "encryptionkey@0"
I20260806 15:54:02.902238 17149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 15:54:02.904733 17175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:02.906086 17149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:02.906237 17149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "fe96b06192d14756aff5ae81f2b640d3"
format_stamp: "Formatted at 2026-08-06 15:54:02 on dist-test-slave-rwrg"
server_key: "6c977c38f0f514331495d1a1b241a11a"
server_key_iv: "cc24a79fadddc971b2a241af0538eae8"
server_key_version: "encryptionkey@0"
I20260806 15:54:02.906360 17149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:02.923269 17149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:02.926621 17149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:02.926856 17149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:02.928617 17149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:02.928679 17149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:02.928761 17149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:02.928807 17149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:54:02 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031642, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:02.941079 17149 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:42227
I20260806 15:54:02.941277 17342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:42227 every 8 connection(s)
I20260806 15:54:02.942152 17149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:02.944329 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:02.929416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:33773 (local address 127.13.223.62:43745)
0806 15:54:02.929605 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:02.929613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:02.930885 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 15:54:02.932189 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:02.932195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:02.932276 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:02.932416 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:02.933589 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.934300 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.935773 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.936027 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.936185 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:02.940423 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:02.940427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:02.940430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:02.940463 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:02.942261 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:02.942895 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:02.942900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:02.942904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:02.942998 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:02.943367 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:02.943369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:02.943370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:02.943552 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:02.943653 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:02.944094 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 15:54:02.944208 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 15:54:02.945178 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:02.930269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.13.223.1:33773)
0806 15:54:02.930729 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:02.930782 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:02.931869 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:02.932624 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:02.932633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:02.932970 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:02.933434 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:02.933447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.934464 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.934467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:02.935646 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:02.935661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.936210 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.936215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:02.936457 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:02.937277 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:02.937301 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:02.940225 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:02.942391 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:02.942397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:02.942412 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:02.942766 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:02.943101 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:02.943105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:02.943107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:02.943277 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:02.943666 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:02.943671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:02.943892 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 15:54:02.944046 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 15:54:02.944297 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
I20260806 15:54:02.946715 17343 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43745
I20260806 15:54:02.946923 17343 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:02.946956 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17149
I20260806 15:54:02.947081 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:02.947396 14204 external_mini_cluster.cc:1428] Setting key 46bd5612dadf3e193ebffb8b986b8b30
I20260806 15:54:02.947628 17343 heartbeater.cc:511] Master 127.13.223.62:43745 requested a full tablet report, sending...
I20260806 15:54:02.949663 17040 ts_manager.cc:194] Registered new tserver with Master: fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227)
I20260806 15:54:02.950769 17040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:33773
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:54:02.960552 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:02.951807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:35327 (local address 127.13.223.62:43745)
0806 15:54:02.951957 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:02.951964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:02.951983 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:54:02.952031 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:02.952036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:02.952096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:02.952199 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:02.953026 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.953512 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.955051 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.955259 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.955336 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:02.956905 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:02.956908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:02.956910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:02.956942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:02.958490 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:02.959122 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:02.959125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:02.959126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:02.959207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:02.959608 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:02.959611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:02.959612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:02.959790 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:02.959870 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:02.960268 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 15:54:02.960368 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:54:02.960769 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:02.949199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.13.223.1:35327)
0806 15:54:02.949517 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:02.949537 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:02.949641 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:02.952359 (+ 2718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:02.952365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:02.952696 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:02.952881 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:02.952893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.953672 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.953679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:02.954884 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:02.954900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:02.955391 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:02.955397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:02.955544 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:02.956112 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:02.956135 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:02.956740 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:02.958644 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:02.958649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:02.958653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:02.958973 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:02.959332 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:02.959336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:02.959339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:02.959465 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:02.959913 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:02.959920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:02.960071 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 15:54:02.960226 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 15:54:02.960400 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:02.978974 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43745
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:03.097890 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:03.098191 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:03.098271 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:03.102252 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:03.102339 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:03.102414 17360 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 15:54:03.102460 17360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:03.102533 17360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:03.107662 17360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:03.108727 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:03.109797 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:03.119644 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:03.119655 17379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:03.119814 17360 server_base.cc:1034] running on GCE node
W20260806 15:54:03.119638 17376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:03.120368 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:03.121083 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:03.122282 17360 hybrid_clock.cc:648] HybridClock initialized: now 1786031643122255 us; error 39 us; skew 500 ppm
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031643, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:03.125066 17360 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:03.125854 17360 webserver.cc:460] Webserver started at http://127.13.223.2:33063/ using document root <none> and password file <none>
I20260806 15:54:03.126372 17360 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:03.126430 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:03.126592 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:03.128618 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "754db59a3f764a579703a95ffe8bb371"
format_stamp: "Formatted at 2026-08-06 15:54:03 on dist-test-slave-rwrg"
server_key: "8cf30123514aad85d960aa67dd6a3959"
server_key_iv: "508dfabcafdebac113328c9a62e43921"
server_key_version: "encryptionkey@0"
I20260806 15:54:03.129150 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "754db59a3f764a579703a95ffe8bb371"
format_stamp: "Formatted at 2026-08-06 15:54:03 on dist-test-slave-rwrg"
server_key: "8cf30123514aad85d960aa67dd6a3959"
server_key_iv: "508dfabcafdebac113328c9a62e43921"
server_key_version: "encryptionkey@0"
I20260806 15:54:03.132822 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:03.135174 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:03.136370 17360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:03.136570 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "754db59a3f764a579703a95ffe8bb371"
format_stamp: "Formatted at 2026-08-06 15:54:03 on dist-test-slave-rwrg"
server_key: "8cf30123514aad85d960aa67dd6a3959"
server_key_iv: "508dfabcafdebac113328c9a62e43921"
server_key_version: "encryptionkey@0"
I20260806 15:54:03.136679 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:03.156177 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:03.159133 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:03.159346 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:03.160959 17360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:03.161010 17360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:03.161051 17360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:03.161067 17360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031643, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:03.172636 17360 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:44721
I20260806 15:54:03.173007 17584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:44721 every 8 connection(s)
I20260806 15:54:03.173658 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:03.175791 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.161824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:34487 (local address 127.13.223.62:43745)
0806 15:54:03.161982 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.161986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.163041 (+ 1055us) server_negotiation.cc:407] Connection header received
0806 15:54:03.164111 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.164114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.164183 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.164249 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:03.166089 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.166734 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.167948 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.168265 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.168381 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.171699 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:03.171702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:03.171705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:03.171739 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:03.173602 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.174195 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.174198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.174200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.174290 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.174687 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.174689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.174690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.174914 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:03.174994 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.175457 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.175610 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:54:03.175945 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17360
I20260806 15:54:03.176069 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:03.176350 14204 external_mini_cluster.cc:1428] Setting key a6d92b097b6087aff34a804df7401373
I20260806 15:54:03.176628 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.162366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.13.223.2:34487)
0806 15:54:03.162874 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.162922 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.163850 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.164640 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.164665 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.165217 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:03.165928 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.165952 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.166859 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.166863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.167822 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.167831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.168459 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.168464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.168747 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.169431 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:03.169451 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:03.171519 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:03.173720 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.173726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.173742 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.174072 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.174382 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.174385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.174387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.174579 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.175009 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:03.175014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:03.175252 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.175407 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.175651 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260806 15:54:03.177884 17585 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43745
I20260806 15:54:03.178030 17585 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:03.178467 17585 heartbeater.cc:511] Master 127.13.223.62:43745 requested a full tablet report, sending...
I20260806 15:54:03.179548 17039 ts_manager.cc:194] Registered new tserver with Master: 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721)
I20260806 15:54:03.180351 17039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:34487
I20260806 15:54:03.188382 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.180636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.13.223.2:44367)
0806 15:54:03.180988 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.181004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.181114 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.181371 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.181374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.181582 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:03.181695 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.181703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.182708 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.182710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.183245 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.183251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.183923 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.183926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.183994 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.184625 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:03.184637 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:03.184947 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:03.186872 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.186875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.186876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.187077 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.187387 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.187389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.187390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.187462 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.187970 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:03.187972 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:03.188063 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.188142 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.188245 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
I20260806 15:54:03.188540 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.180435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:44367 (local address 127.13.223.62:43745)
0806 15:54:03.180668 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.180672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.181090 (+ 418us) server_negotiation.cc:407] Connection header received
0806 15:54:03.181194 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.181198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.181250 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.181338 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:03.181835 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.182561 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.183363 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.183781 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.183906 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.185101 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:03.185107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:03.185113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:03.185146 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:03.186760 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.187182 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.187186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.187189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.187290 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.187550 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.187553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.187555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.187862 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:03.187973 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.188239 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.188357 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:03.205471 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43745
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:03.318104 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:03.318374 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:03.318470 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:03.322480 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:03.322574 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:03.322646 17599 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 15:54:03.322690 17599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:03.322752 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:03.327821 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:03.328902 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:03.329939 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:03.339918 17611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:03.340003 17614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:03.340081 17599 server_base.cc:1034] running on GCE node
W20260806 15:54:03.339975 17612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:03.340792 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:03.341531 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:03.342720 17599 hybrid_clock.cc:648] HybridClock initialized: now 1786031643342703 us; error 29 us; skew 500 ppm
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031643, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:03.345654 17599 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:03.346699 17599 webserver.cc:460] Webserver started at http://127.13.223.3:46497/ using document root <none> and password file <none>
I20260806 15:54:03.347283 17599 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:03.347389 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:03.347626 17599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:03.350319 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "dc67b3f5d00848f391ba6f784174d284"
format_stamp: "Formatted at 2026-08-06 15:54:03 on dist-test-slave-rwrg"
server_key: "b966dc7f2345ceb49cf089ec17ecb339"
server_key_iv: "b4a737b859f982ddf8db3e026bfccae8"
server_key_version: "encryptionkey@0"
I20260806 15:54:03.351022 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "dc67b3f5d00848f391ba6f784174d284"
format_stamp: "Formatted at 2026-08-06 15:54:03 on dist-test-slave-rwrg"
server_key: "b966dc7f2345ceb49cf089ec17ecb339"
server_key_iv: "b4a737b859f982ddf8db3e026bfccae8"
server_key_version: "encryptionkey@0"
I20260806 15:54:03.355326 17599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:03.358160 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:03.359540 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:54:03.359687 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "dc67b3f5d00848f391ba6f784174d284"
format_stamp: "Formatted at 2026-08-06 15:54:03 on dist-test-slave-rwrg"
server_key: "b966dc7f2345ceb49cf089ec17ecb339"
server_key_iv: "b4a737b859f982ddf8db3e026bfccae8"
server_key_version: "encryptionkey@0"
I20260806 15:54:03.359808 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:03.370205 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:03.373478 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:03.373711 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:03.375345 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:03.375392 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:03.375432 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:03.375488 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031643, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:03.391008 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.376289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:60433 (local address 127.13.223.62:43745)
0806 15:54:03.376450 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.376454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.377717 (+ 1263us) server_negotiation.cc:407] Connection header received
0806 15:54:03.378728 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.378731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.378793 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.378918 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:03.380460 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.381139 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.382446 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.382684 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.382802 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.386090 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:03.386094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:03.386096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:03.386130 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:03.388706 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.389364 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.389369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.389370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.389461 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.389873 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.389876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.389878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.390072 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:03.390194 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.390708 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.390832 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 15:54:03.391906 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.376859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.13.223.3:60433)
0806 15:54:03.377548 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.377599 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.378479 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.379131 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.379142 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.379617 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:03.380289 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.380313 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.381269 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.381272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.382326 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.382339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.382842 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.382845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.383054 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.383685 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:03.383701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:03.385916 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:03.388912 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.388919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.388936 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.389225 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.389563 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.389567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.389570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.389758 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.390193 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:03.390200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:03.390461 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.390662 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.390979 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":264,"threads_started":1}
I20260806 15:54:03.394766 17599 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40829
I20260806 15:54:03.394785 17832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40829 every 8 connection(s)
I20260806 15:54:03.395826 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:03.397401 17836 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43745
I20260806 15:54:03.397547 17836 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:03.398007 17836 heartbeater.cc:511] Master 127.13.223.62:43745 requested a full tablet report, sending...
I20260806 15:54:03.399129 17039 ts_manager.cc:194] Registered new tserver with Master: dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:03.399766 17039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:60433
I20260806 15:54:03.402777 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17599
I20260806 15:54:03.402938 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:03.403287 14204 external_mini_cluster.cc:1428] Setting key 934cf655096fe49eb6daa3c63dc69913
I20260806 15:54:03.403870 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.395722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:58583 (local address 127.13.223.62:43745)
0806 15:54:03.395879 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.395885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.395975 (+ 90us) server_negotiation.cc:407] Connection header received
0806 15:54:03.396080 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.396087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.396139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.396249 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:03.396784 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.397499 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.398675 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.399030 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.399169 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.400476 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:03.400479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:03.400482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:03.400580 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:03.402143 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.402589 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.402592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.402594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.402724 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.403046 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.403049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.403050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.403235 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:03.403345 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.403614 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.403725 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 15:54:03.403870 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.395641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.13.223.3:58583)
0806 15:54:03.395891 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.395909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.396010 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.396250 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.396252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.396463 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:03.396644 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.396653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.397660 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.397663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.398543 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.398559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.399176 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.399179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.399305 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.399883 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:03.399899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:03.400306 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:03.402260 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.402263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.402264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.402452 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.402832 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.402835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.402837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.402950 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.403337 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:03.403340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:03.403437 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.403546 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.403686 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260806 15:54:03.405694 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:03.435768 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.426828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.13.223.62:43745)
0806 15:54:03.426962 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.426965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.426979 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:03.427023 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.427025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.427067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.427140 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:03.427892 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.428414 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.429380 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.429614 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.429695 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.432115 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:03.432118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:03.432120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:03.432149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:03.434019 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.434427 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.434430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.434431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.434553 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.434808 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.434810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.434811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.434999 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:03.435120 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.435340 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.435454 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:54:03.439064 17038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:03.442559 17038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:03.457324 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.451399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44721 (local address 127.0.0.1:54538)
0806 15:54:03.451862 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.451910 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.452069 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.452889 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.452895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.452919 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.453066 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.453073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.454257 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.454262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.455309 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.455320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.456743 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.456746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.456859 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.456900 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.457031 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.457168 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":88,"threads_started":1}
I20260806 15:54:03.457325 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.451558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.13.223.2:44721)
0806 15:54:03.451959 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.451996 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.452018 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:54:03.452296 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.452302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.452542 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.452708 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.453224 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.454139 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.455454 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.456621 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.456722 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.456884 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.457077 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.457172 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":107,"threads_started":1}
I20260806 15:54:03.457495 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.452153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42227 (local address 127.0.0.1:57434)
0806 15:54:03.452456 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.452469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.452631 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.453305 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.453308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.453327 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.453448 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.453459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.454981 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.454984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.455976 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.455986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.457147 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.457151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.457239 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.457261 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.457308 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.457380 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260806 15:54:03.458069 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.452332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.13.223.1:42227)
0806 15:54:03.452872 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.452878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.452900 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:54:03.452978 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.452985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.453164 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.453614 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.453749 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.454850 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.456132 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.457004 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.457693 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.457820 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.457861 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.457926 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":91,"threads_started":1}
I20260806 15:54:03.458500 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.451497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40829 (local address 127.0.0.1:43632)
0806 15:54:03.452157 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.452170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.452258 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.453654 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.453657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.453674 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.453767 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.453776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.455237 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.455241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.456346 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.456359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.458095 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.458099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.458219 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.458243 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.458311 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.458377 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":125,"threads_started":1}
I20260806 15:54:03.458811 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.452601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.13.223.3:40829)
0806 15:54:03.452938 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.452948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.452967 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:54:03.453199 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.453207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.453373 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.453913 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.454059 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.455125 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.456516 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.457332 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.458068 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.458375 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.458429 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.458517 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260806 15:54:03.460711 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 122ced30c59240568cd24c46a32bf515 (DEFAULT_TABLE table=test-table [id=244f79ec9b5b4fe487a98055bafe9ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:03.460788 17253 tablet_service.cc:1467] Processing CreateTablet for tablet 122ced30c59240568cd24c46a32bf515 (DEFAULT_TABLE table=test-table [id=244f79ec9b5b4fe487a98055bafe9ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:03.461336 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 122ced30c59240568cd24c46a32bf515 (DEFAULT_TABLE table=test-table [id=244f79ec9b5b4fe487a98055bafe9ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:03.461794 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122ced30c59240568cd24c46a32bf515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:03.461865 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122ced30c59240568cd24c46a32bf515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:03.462227 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122ced30c59240568cd24c46a32bf515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:03.469303 17864 tablet_bootstrap.cc:492] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Bootstrap starting.
I20260806 15:54:03.469868 17868 tablet_bootstrap.cc:492] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Bootstrap starting.
I20260806 15:54:03.470120 17870 tablet_bootstrap.cc:492] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Bootstrap starting.
I20260806 15:54:03.471637 17864 tablet_bootstrap.cc:654] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:03.471839 17868 tablet_bootstrap.cc:654] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:03.471976 17870 tablet_bootstrap.cc:654] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:03.472572 17868 log.cc:824] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:03.472559 17864 log.cc:824] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:03.472780 17870 log.cc:824] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:03.474710 17870 tablet_bootstrap.cc:492] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: No bootstrap required, opened a new log
I20260806 15:54:03.474709 17864 tablet_bootstrap.cc:492] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: No bootstrap required, opened a new log
I20260806 15:54:03.474722 17868 tablet_bootstrap.cc:492] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: No bootstrap required, opened a new log
I20260806 15:54:03.474936 17870 ts_tablet_manager.cc:1397] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 15:54:03.474936 17864 ts_tablet_manager.cc:1397] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:54:03.474952 17868 ts_tablet_manager.cc:1397] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 15:54:03.478152 17868 raft_consensus.cc:348] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.478446 17868 raft_consensus.cc:374] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:03.478538 17868 raft_consensus.cc:729] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe96b06192d14756aff5ae81f2b640d3, State: Initialized, Role: FOLLOWER
I20260806 15:54:03.478837 17864 raft_consensus.cc:348] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.479086 17864 raft_consensus.cc:374] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:03.478960 17870 raft_consensus.cc:348] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.479024 17868 consensus_queue.cc:260] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.479140 17864 raft_consensus.cc:729] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Initialized, Role: FOLLOWER
I20260806 15:54:03.479174 17870 raft_consensus.cc:374] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:03.479224 17870 raft_consensus.cc:729] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc67b3f5d00848f391ba6f784174d284, State: Initialized, Role: FOLLOWER
I20260806 15:54:03.479591 17864 consensus_queue.cc:260] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.479677 17343 heartbeater.cc:503] Master 127.13.223.62:43745 was elected leader, sending a full tablet report...
I20260806 15:54:03.480229 17868 ts_tablet_manager.cc:1428] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 15:54:03.480283 17585 heartbeater.cc:503] Master 127.13.223.62:43745 was elected leader, sending a full tablet report...
I20260806 15:54:03.480522 17864 ts_tablet_manager.cc:1428] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260806 15:54:03.480787 17870 consensus_queue.cc:260] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.481756 17870 ts_tablet_manager.cc:1428] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260806 15:54:03.482074 17836 heartbeater.cc:503] Master 127.13.223.62:43745 was elected leader, sending a full tablet report...
I20260806 15:54:03.515892 17881 raft_consensus.cc:482] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:03.516155 17881 raft_consensus.cc:504] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.517344 17881 leader_election.cc:288] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829), fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227)
I20260806 15:54:03.521118 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.517683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40829 (local address 127.13.223.2:55203)
0806 15:54:03.517825 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.517858 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.517957 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.518274 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.518277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.518297 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.518428 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.518436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.519193 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.519197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.519965 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.519974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.520688 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.520692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.520793 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.520818 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.520907 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.520981 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 15:54:03.521112 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.517803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:55203 (local address 127.13.223.3:40829)
0806 15:54:03.517940 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.517944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.517959 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:03.518096 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.518099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.518148 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.518239 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.518551 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.519078 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.520090 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.520562 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.520698 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.520750 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.520924 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.520992 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:54:03.521814 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122ced30c59240568cd24c46a32bf515" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284" is_pre_election: true
I20260806 15:54:03.522159 17756 raft_consensus.cc:2454] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:03.522400 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.518227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:37457 (local address 127.13.223.1:42227)
0806 15:54:03.518441 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.518445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.518631 (+ 186us) server_negotiation.cc:407] Connection header received
0806 15:54:03.518723 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.518726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.518779 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.518870 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.519588 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.520130 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.521399 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.521854 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.521999 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.522047 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.522227 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.522297 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 15:54:03.522413 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.518042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42227 (local address 127.13.223.2:37457)
0806 15:54:03.518467 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.518480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.518623 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.518883 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.518886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.518906 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.519430 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.519438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.520259 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.520265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.521152 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.521186 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.521997 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.522001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.522094 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.522114 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.522211 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.522276 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":155,"threads_started":1}
I20260806 15:54:03.522701 17395 leader_election.cc:302] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, dc67b3f5d00848f391ba6f784174d284; no voters:
I20260806 15:54:03.522935 17881 raft_consensus.cc:2781] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:03.523048 17881 raft_consensus.cc:482] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:03.523124 17881 raft_consensus.cc:3037] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:03.523355 17281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122ced30c59240568cd24c46a32bf515" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3" is_pre_election: true
I20260806 15:54:03.523630 17281 raft_consensus.cc:2454] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:03.524533 17881 raft_consensus.cc:504] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.524946 17881 leader_election.cc:288] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Requested vote from peers dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829), fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227)
I20260806 15:54:03.525296 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122ced30c59240568cd24c46a32bf515" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284"
I20260806 15:54:03.525408 17281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122ced30c59240568cd24c46a32bf515" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3"
I20260806 15:54:03.525434 17756 raft_consensus.cc:3037] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:03.525547 17281 raft_consensus.cc:3037] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:03.526669 17756 raft_consensus.cc:2454] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:03.526865 17281 raft_consensus.cc:2454] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:03.527034 17395 leader_election.cc:302] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, dc67b3f5d00848f391ba6f784174d284; no voters:
I20260806 15:54:03.527274 17881 raft_consensus.cc:2781] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:03.527508 17881 raft_consensus.cc:686] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Becoming Leader. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Running, Role: LEADER
I20260806 15:54:03.527902 17881 consensus_queue.cc:237] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } }
I20260806 15:54:03.531365 17039 catalog_manager.cc:5495] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754db59a3f764a579703a95ffe8bb371 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "754db59a3f764a579703a95ffe8bb371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } health_report { overall_health: UNKNOWN } } }
Aug 06 15:54:03 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031642, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:03.583491 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.574613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.13.223.62:43745)
0806 15:54:03.574765 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.574770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.575002 (+ 232us) server_negotiation.cc:407] Connection header received
0806 15:54:03.575192 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.575195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.575242 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.575325 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:03.575998 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.576618 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.577449 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.577690 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.577799 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.580311 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:03.580314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:03.580316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:03.580340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:03.581779 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.582230 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.582236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.582238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.582314 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:03.582591 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:03.582594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:03.582595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:03.582787 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:03.582896 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.583191 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.583356 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":46}
I20260806 15:54:03.583491 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.574564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43745 (local address 127.0.0.1:60398)
0806 15:54:03.574905 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.574918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.575084 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.575351 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.575354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.575783 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:03.575876 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.575883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.576727 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.576733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.577326 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.577345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.577803 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.577806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.577880 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.578542 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:03.578554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:03.580180 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:03.581886 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.581899 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.581905 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.582130 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.582410 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:03.582413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:03.582415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:03.582496 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:03.582896 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:03.582905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:03.583015 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.583134 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.583300 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260806 15:54:03.587824 17039 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:60398:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 15:54:03.588481 17039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:03.593605 17253 tablet_service.cc:1467] Processing CreateTablet for tablet 644a2ab9746d489782b1c407305ba564 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a7ff3a5f23542939a401fd4b96ba5d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:03.593626 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 644a2ab9746d489782b1c407305ba564 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a7ff3a5f23542939a401fd4b96ba5d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:03.594419 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644a2ab9746d489782b1c407305ba564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:03.594419 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644a2ab9746d489782b1c407305ba564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:03.594065 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 644a2ab9746d489782b1c407305ba564 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a7ff3a5f23542939a401fd4b96ba5d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:03.594743 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644a2ab9746d489782b1c407305ba564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:03.597229 17870 tablet_bootstrap.cc:492] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: Bootstrap starting.
I20260806 15:54:03.597301 17864 tablet_bootstrap.cc:492] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: Bootstrap starting.
I20260806 15:54:03.597301 17868 tablet_bootstrap.cc:492] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: Bootstrap starting.
I20260806 15:54:03.598122 17870 tablet_bootstrap.cc:654] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:03.598285 17868 tablet_bootstrap.cc:654] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:03.598285 17864 tablet_bootstrap.cc:654] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:03.599638 17868 tablet_bootstrap.cc:492] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: No bootstrap required, opened a new log
I20260806 15:54:03.599663 17864 tablet_bootstrap.cc:492] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: No bootstrap required, opened a new log
I20260806 15:54:03.599752 17868 ts_tablet_manager.cc:1397] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:03.599766 17864 ts_tablet_manager.cc:1397] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:03.600181 17868 raft_consensus.cc:348] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.600184 17864 raft_consensus.cc:348] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.600298 17864 raft_consensus.cc:374] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:03.600302 17868 raft_consensus.cc:374] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:03.600342 17864 raft_consensus.cc:729] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Initialized, Role: FOLLOWER
I20260806 15:54:03.600349 17868 raft_consensus.cc:729] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe96b06192d14756aff5ae81f2b640d3, State: Initialized, Role: FOLLOWER
I20260806 15:54:03.600526 17868 consensus_queue.cc:260] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.600531 17864 consensus_queue.cc:260] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.600841 17868 ts_tablet_manager.cc:1428] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:03.600831 17864 ts_tablet_manager.cc:1428] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:03.600813 17879 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.600836 17881 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.600965 17879 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:03.600988 17881 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:03.601974 17870 tablet_bootstrap.cc:492] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: No bootstrap required, opened a new log
I20260806 15:54:03.602058 17870 ts_tablet_manager.cc:1397] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260806 15:54:03.602620 17870 raft_consensus.cc:348] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.602798 17870 raft_consensus.cc:374] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:03.602835 17870 raft_consensus.cc:729] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc67b3f5d00848f391ba6f784174d284, State: Initialized, Role: FOLLOWER
I20260806 15:54:03.602975 17870 consensus_queue.cc:260] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.603267 17870 ts_tablet_manager.cc:1428] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 15:54:03.603502 17882 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.603662 17882 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 15:54:03.647310 17837 tablet.cc:2369] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:03.674896 17586 tablet.cc:2369] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:03.693670 17345 tablet.cc:2369] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:03.866636 17879 raft_consensus.cc:482] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:03.866850 17879 raft_consensus.cc:504] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.868126 17879 leader_election.cc:288] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:03.873912 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.868637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:50615 (local address 127.13.223.2:44721)
0806 15:54:03.868755 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.868760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.869066 (+ 306us) server_negotiation.cc:407] Connection header received
0806 15:54:03.869217 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.869220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.869284 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.869367 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.869922 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.870592 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.872247 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.873059 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.873227 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.873296 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.873660 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.873752 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:54:03.873912 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.868594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:45631 (local address 127.13.223.3:40829)
0806 15:54:03.868754 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.868760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.869083 (+ 323us) server_negotiation.cc:407] Connection header received
0806 15:54:03.869213 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.869216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.869284 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.869367 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:03.870319 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.870993 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.872376 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.873086 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.873227 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.873294 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.873660 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.873752 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 15:54:03.873991 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.868459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40829 (local address 127.13.223.1:45631)
0806 15:54:03.868977 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.868991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.869092 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.869397 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.869401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.869436 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.870188 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.870197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.871123 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.871127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.872229 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.872237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.873233 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.873237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.873466 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.873524 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.873609 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.873761 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":206,"spinlock_wait_cycles":20480,"thread_start_us":180,"threads_started":1}
I20260806 15:54:03.874197 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.868451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44721 (local address 127.13.223.1:50615)
0806 15:54:03.868922 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.868938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.869102 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.869397 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.869401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.869437 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:03.869787 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.869799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.870798 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.870801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.872087 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.872098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.873233 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.873237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.873462 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.873524 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.873609 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.873761 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":275,"spinlock_wait_cycles":49664,"thread_start_us":204,"threads_started":1}
I20260806 15:54:03.874645 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644a2ab9746d489782b1c407305ba564" candidate_uuid: "fe96b06192d14756aff5ae81f2b640d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284" is_pre_election: true
I20260806 15:54:03.874963 17756 raft_consensus.cc:2454] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe96b06192d14756aff5ae81f2b640d3 in term 0.
I20260806 15:54:03.875115 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644a2ab9746d489782b1c407305ba564" candidate_uuid: "fe96b06192d14756aff5ae81f2b640d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754db59a3f764a579703a95ffe8bb371" is_pre_election: true
I20260806 15:54:03.875401 17510 raft_consensus.cc:2454] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe96b06192d14756aff5ae81f2b640d3 in term 0.
I20260806 15:54:03.875965 17176 leader_election.cc:302] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, fe96b06192d14756aff5ae81f2b640d3; no voters:
I20260806 15:54:03.876315 17879 raft_consensus.cc:2781] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:03.876379 17879 raft_consensus.cc:482] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:03.876406 17879 raft_consensus.cc:3037] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:03.877446 17879 raft_consensus.cc:504] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.877866 17879 leader_election.cc:288] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [CANDIDATE]: Term 1 election: Requested vote from peers 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:03.878225 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644a2ab9746d489782b1c407305ba564" candidate_uuid: "fe96b06192d14756aff5ae81f2b640d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754db59a3f764a579703a95ffe8bb371"
I20260806 15:54:03.878360 17510 raft_consensus.cc:3037] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:03.878332 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644a2ab9746d489782b1c407305ba564" candidate_uuid: "fe96b06192d14756aff5ae81f2b640d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284"
I20260806 15:54:03.878469 17756 raft_consensus.cc:3037] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:03.879530 17756 raft_consensus.cc:2454] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe96b06192d14756aff5ae81f2b640d3 in term 1.
I20260806 15:54:03.879639 17510 raft_consensus.cc:2454] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe96b06192d14756aff5ae81f2b640d3 in term 1.
I20260806 15:54:03.879843 17180 leader_election.cc:302] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc67b3f5d00848f391ba6f784174d284, fe96b06192d14756aff5ae81f2b640d3; no voters:
I20260806 15:54:03.880064 17879 raft_consensus.cc:2781] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:03.880357 17879 raft_consensus.cc:686] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [term 1 LEADER]: Becoming Leader. State: Replica: fe96b06192d14756aff5ae81f2b640d3, State: Running, Role: LEADER
I20260806 15:54:03.880748 17879 consensus_queue.cc:237] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:03.883183 17898 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: TxnStatusTablet state changed. Reason: New leader fe96b06192d14756aff5ae81f2b640d3. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.883302 17898 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:03.883633 17901 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 15:54:03.883848 17901 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 15:54:03.884284 17038 catalog_manager.cc:5495] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1). New cstate: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:03.902479 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.898273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.13.223.1:42227)
0806 15:54:03.898391 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.898396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.898504 (+ 108us) server_negotiation.cc:407] Connection header received
0806 15:54:03.898650 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.898654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.898718 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.898831 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:03.899149 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.899937 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.900820 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.901167 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.901274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.901546 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.902054 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.902164 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.902273 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.902333 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:54:03.902524 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.898155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42227 (local address 127.0.0.1:57446)
0806 15:54:03.898423 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.898436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.898532 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.898892 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.898895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.898902 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:03.899015 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.899021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.900043 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.900046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.900709 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.900718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.901279 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.901281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.901352 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.901397 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.902152 (+ 755us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.902194 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.902258 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.902360 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260806 15:54:03.908100 17510 raft_consensus.cc:1261] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Refusing update from remote peer fe96b06192d14756aff5ae81f2b640d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:03.908102 17756 raft_consensus.cc:1261] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Refusing update from remote peer fe96b06192d14756aff5ae81f2b640d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:03.908824 17898 consensus_queue.cc:1035] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:03.909005 17879 consensus_queue.cc:1035] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:03.912108 17881 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: TxnStatusTablet state changed. Reason: New leader fe96b06192d14756aff5ae81f2b640d3. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.912230 17881 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:03.912894 17882 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: TxnStatusTablet state changed. Reason: New leader fe96b06192d14756aff5ae81f2b640d3. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.913007 17882 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:03.916437 17879 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.916596 17879 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:03.916603 17882 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.916739 17882 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P dc67b3f5d00848f391ba6f784174d284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:03.916811 17904 mvcc.cc:205] Tried to move back new op lower bound from 7315585613440516096 to 7315585613338267648. Current Snapshot: MvccSnapshot[applied={T|T < 7315585613440516096}]
I20260806 15:54:03.917855 17879 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.917968 17879 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P fe96b06192d14756aff5ae81f2b640d3: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:03.919143 17881 tablet_replica.cc:439] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe96b06192d14756aff5ae81f2b640d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } } }
I20260806 15:54:03.919277 17881 tablet_replica.cc:442] T 644a2ab9746d489782b1c407305ba564 P 754db59a3f764a579703a95ffe8bb371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:03.919837 17906 mvcc.cc:205] Tried to move back new op lower bound from 7315585613440516096 to 7315585613338267648. Current Snapshot: MvccSnapshot[applied={T|T < 7315585613440516096}]
I20260806 15:54:03.921267 17905 mvcc.cc:205] Tried to move back new op lower bound from 7315585613440516096 to 7315585613338267648. Current Snapshot: MvccSnapshot[applied={T|T < 7315585613440516096}]
I20260806 15:54:03.930011 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.926059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.13.223.2:44721)
0806 15:54:03.926194 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.926198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.926209 (+ 11us) server_negotiation.cc:407] Connection header received
0806 15:54:03.926269 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.926271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.926340 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.926448 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:03.926715 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.927231 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.928599 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.928818 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.928899 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.929209 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.929598 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.929684 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.929812 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.929860 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:54:03.941773 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.937799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:40245 (local address 127.13.223.1:42227)
0806 15:54:03.937950 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.937955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.938151 (+ 196us) server_negotiation.cc:407] Connection header received
0806 15:54:03.938246 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.938249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.938300 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.938382 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:03.938617 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.939426 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.940358 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.940750 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.940853 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.941274 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.941361 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.941407 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.941588 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.941643 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 15:54:03.942183 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.937601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42227 (local address 127.13.223.2:40245)
0806 15:54:03.938068 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.938081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.938170 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.938397 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.938399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.938405 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:03.938502 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.938506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.939549 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.939551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.940250 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.940261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.940937 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.940941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.941041 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.941093 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.941468 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.941473 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.941940 (+ 467us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.942025 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":288,"threads_started":1}
I20260806 15:54:03.950765 17281 raft_consensus.cc:1261] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:03.951344 17881 consensus_queue.cc:1035] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:03.951774 17756 raft_consensus.cc:1261] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:03.952163 17881 consensus_queue.cc:1035] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:03.982193 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.976569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:40521 (local address 127.13.223.2:44721)
0806 15:54:03.976722 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:03.976727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:03.977239 (+ 512us) server_negotiation.cc:407] Connection header received
0806 15:54:03.977562 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:03.977569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:03.977634 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:03.977744 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:03.978335 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.979119 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.980659 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.980970 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.981075 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.981595 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.981704 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.981811 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:03.981972 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 15:54:03.982035 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:54:03.982229 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:03.976465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44721 (local address 127.13.223.1:40521)
0806 15:54:03.976996 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:03.977012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:03.977279 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:03.977756 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:03.977759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:03.977846 (+ 87us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:03.978174 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.978187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.979322 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.979326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.980529 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:03.980542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:03.981224 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:03.981228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:03.981395 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:03.981448 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:03.981834 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:03.981842 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 15:54:03.981951 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 15:54:03.982035 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":239,"threads_started":1}
I20260806 15:54:04.011617 17040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:04.011822 17040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:04.014658 17040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Sending DeleteTablet for 3 replicas of tablet 122ced30c59240568cd24c46a32bf515
I20260806 15:54:04.015457 17479 tablet_service.cc:1514] Processing DeleteTablet for tablet 122ced30c59240568cd24c46a32bf515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:04 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:54538
I20260806 15:54:04.015465 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet 122ced30c59240568cd24c46a32bf515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:04 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:43632
I20260806 15:54:04.016167 17934 tablet_replica.cc:330] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: stopping tablet replica
I20260806 15:54:04.016388 17253 tablet_service.cc:1514] Processing DeleteTablet for tablet 122ced30c59240568cd24c46a32bf515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:04 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:57434
I20260806 15:54:04.016675 17934 raft_consensus.cc:2229] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:04.017190 17934 raft_consensus.cc:2258] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:04.017809 17936 tablet_replica.cc:330] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: stopping tablet replica
I20260806 15:54:04.017765 17039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 15:54:04.018136 17936 raft_consensus.cc:2229] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 15:54:04.018213 17039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:04.018354 17936 raft_consensus.cc:2258] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:04.019001 17934 ts_tablet_manager.cc:1905] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:04.019627 17936 ts_tablet_manager.cc:1905] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:04.022864 17937 tablet_replica.cc:330] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: stopping tablet replica
I20260806 15:54:04.023463 17253 tablet_service.cc:1467] Processing CreateTablet for tablet c8f8eebdfff8424598efe7ff054fe741 (DEFAULT_TABLE table=test-table [id=0ddd923da8ec46d6b1b3e868f9f3c33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:04.023554 17479 tablet_service.cc:1467] Processing CreateTablet for tablet c8f8eebdfff8424598efe7ff054fe741 (DEFAULT_TABLE table=test-table [id=0ddd923da8ec46d6b1b3e868f9f3c33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:04.023792 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f8eebdfff8424598efe7ff054fe741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:04.023818 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f8eebdfff8424598efe7ff054fe741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:04.026311 17864 tablet_bootstrap.cc:492] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Bootstrap starting.
I20260806 15:54:04.026459 17868 tablet_bootstrap.cc:492] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Bootstrap starting.
I20260806 15:54:04.027290 17864 tablet_bootstrap.cc:654] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:04.027386 17717 tablet_service.cc:1467] Processing CreateTablet for tablet c8f8eebdfff8424598efe7ff054fe741 (DEFAULT_TABLE table=test-table [id=0ddd923da8ec46d6b1b3e868f9f3c33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:04.027495 17868 tablet_bootstrap.cc:654] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:04.027676 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f8eebdfff8424598efe7ff054fe741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:04.028453 17937 raft_consensus.cc:2229] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:04.029134 17934 ts_tablet_manager.cc:1918] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:04.029417 17937 raft_consensus.cc:2258] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:04.029712 17934 log.cc:1196] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/122ced30c59240568cd24c46a32bf515
I20260806 15:54:04.030256 17870 tablet_bootstrap.cc:492] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Bootstrap starting.
I20260806 15:54:04.030697 17934 ts_tablet_manager.cc:1939] T 122ced30c59240568cd24c46a32bf515 P 754db59a3f764a579703a95ffe8bb371: Deleting consensus metadata
I20260806 15:54:04.031298 17870 tablet_bootstrap.cc:654] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:04.032581 17017 catalog_manager.cc:4841] TS 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721): tablet 122ced30c59240568cd24c46a32bf515 (table test-table [id=244f79ec9b5b4fe487a98055bafe9ea8]) successfully deleted
I20260806 15:54:04.031329 17937 ts_tablet_manager.cc:1905] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:04.035665 17937 ts_tablet_manager.cc:1918] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:04.036144 17936 ts_tablet_manager.cc:1918] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:04.036253 17936 log.cc:1196] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/122ced30c59240568cd24c46a32bf515
I20260806 15:54:04.036612 17864 tablet_bootstrap.cc:492] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: No bootstrap required, opened a new log
I20260806 15:54:04.036701 17864 ts_tablet_manager.cc:1397] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260806 15:54:04.036746 17936 ts_tablet_manager.cc:1939] T 122ced30c59240568cd24c46a32bf515 P dc67b3f5d00848f391ba6f784174d284: Deleting consensus metadata
I20260806 15:54:04.037179 17864 raft_consensus.cc:348] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.037293 17864 raft_consensus.cc:374] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:04.037322 17864 raft_consensus.cc:729] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Initialized, Role: FOLLOWER
I20260806 15:54:04.037439 17864 consensus_queue.cc:260] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.037676 17864 ts_tablet_manager.cc:1428] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:04.038965 17870 tablet_bootstrap.cc:492] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: No bootstrap required, opened a new log
I20260806 15:54:04.039067 17870 ts_tablet_manager.cc:1397] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260806 15:54:04.039515 17870 raft_consensus.cc:348] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.039633 17870 raft_consensus.cc:374] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:04.039666 17870 raft_consensus.cc:729] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc67b3f5d00848f391ba6f784174d284, State: Initialized, Role: FOLLOWER
I20260806 15:54:04.039798 17870 consensus_queue.cc:260] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.040077 17870 ts_tablet_manager.cc:1428] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 15:54:04.040162 17937 log.cc:1196] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/122ced30c59240568cd24c46a32bf515
I20260806 15:54:04.040717 17937 ts_tablet_manager.cc:1939] T 122ced30c59240568cd24c46a32bf515 P fe96b06192d14756aff5ae81f2b640d3: Deleting consensus metadata
I20260806 15:54:04.041283 17868 tablet_bootstrap.cc:492] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: No bootstrap required, opened a new log
I20260806 15:54:04.041389 17868 ts_tablet_manager.cc:1397] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260806 15:54:04.041903 17868 raft_consensus.cc:348] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.042034 17868 raft_consensus.cc:374] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:04.042076 17868 raft_consensus.cc:729] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe96b06192d14756aff5ae81f2b640d3, State: Initialized, Role: FOLLOWER
I20260806 15:54:04.042193 17024 catalog_manager.cc:4841] TS fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227): tablet 122ced30c59240568cd24c46a32bf515 (table test-table [id=244f79ec9b5b4fe487a98055bafe9ea8]) successfully deleted
I20260806 15:54:04.042200 17868 consensus_queue.cc:260] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.042474 17868 ts_tablet_manager.cc:1428] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:04.043967 17024 catalog_manager.cc:4841] TS dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829): tablet 122ced30c59240568cd24c46a32bf515 (table test-table [id=244f79ec9b5b4fe487a98055bafe9ea8]) successfully deleted
I20260806 15:54:04.046293 17884 raft_consensus.cc:482] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:04.046401 17884 raft_consensus.cc:504] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.046785 17884 leader_election.cc:288] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:04.047266 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f8eebdfff8424598efe7ff054fe741" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284" is_pre_election: true
I20260806 15:54:04.047252 17281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f8eebdfff8424598efe7ff054fe741" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3" is_pre_election: true
I20260806 15:54:04.047399 17756 raft_consensus.cc:2454] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:04.047408 17281 raft_consensus.cc:2454] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:04.047765 17395 leader_election.cc:302] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, fe96b06192d14756aff5ae81f2b640d3; no voters:
I20260806 15:54:04.048164 17884 raft_consensus.cc:2781] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:04.048238 17884 raft_consensus.cc:482] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:04.048285 17884 raft_consensus.cc:3037] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:04.049707 17884 raft_consensus.cc:504] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.050123 17884 leader_election.cc:288] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Requested vote from peers fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:04.050484 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f8eebdfff8424598efe7ff054fe741" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284"
I20260806 15:54:04.050537 17281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f8eebdfff8424598efe7ff054fe741" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3"
I20260806 15:54:04.050611 17756 raft_consensus.cc:3037] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:04.050647 17281 raft_consensus.cc:3037] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:04.051921 17281 raft_consensus.cc:2454] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:04.052106 17756 raft_consensus.cc:2454] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:04.052366 17395 leader_election.cc:302] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, fe96b06192d14756aff5ae81f2b640d3; no voters:
I20260806 15:54:04.052603 17884 raft_consensus.cc:2781] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:04.052681 17884 raft_consensus.cc:686] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Becoming Leader. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Running, Role: LEADER
I20260806 15:54:04.052819 17884 consensus_queue.cc:237] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:04.054857 17039 catalog_manager.cc:5495] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754db59a3f764a579703a95ffe8bb371 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "754db59a3f764a579703a95ffe8bb371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:04.075798 17756 raft_consensus.cc:1261] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:04.076196 17884 consensus_queue.cc:1035] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:04.076305 17281 raft_consensus.cc:1261] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:04.076879 17884 consensus_queue.cc:1035] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 15:54:04.099013 17906 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 15:54:05.650158 17039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:05.650332 17039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:05.652920 17039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Sending DeleteTablet for 3 replicas of tablet c8f8eebdfff8424598efe7ff054fe741
I20260806 15:54:05.653429 17253 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f8eebdfff8424598efe7ff054fe741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:05 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:57434
I20260806 15:54:05.653491 17479 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f8eebdfff8424598efe7ff054fe741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:05 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:54538
I20260806 15:54:05.653596 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f8eebdfff8424598efe7ff054fe741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:05 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:43632
I20260806 15:54:05.653826 17988 tablet_replica.cc:330] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: stopping tablet replica
I20260806 15:54:05.653962 17988 raft_consensus.cc:2229] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:05.654130 17998 tablet_replica.cc:330] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: stopping tablet replica
I20260806 15:54:05.654291 17998 raft_consensus.cc:2229] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:05.654431 17998 raft_consensus.cc:2258] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:05.655088 17998 ts_tablet_manager.cc:1905] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:05.657769 17998 ts_tablet_manager.cc:1918] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 15:54:05.657899 17998 log.cc:1196] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/c8f8eebdfff8424598efe7ff054fe741
I20260806 15:54:05.658227 17998 ts_tablet_manager.cc:1939] T c8f8eebdfff8424598efe7ff054fe741 P dc67b3f5d00848f391ba6f784174d284: Deleting consensus metadata
I20260806 15:54:05.658922 17024 catalog_manager.cc:4841] TS dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829): tablet c8f8eebdfff8424598efe7ff054fe741 (table test-table [id=0ddd923da8ec46d6b1b3e868f9f3c33e]) successfully deleted
I20260806 15:54:05.654098 17988 raft_consensus.cc:2258] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:05.660810 17995 tablet_replica.cc:330] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: stopping tablet replica
I20260806 15:54:05.660938 17995 raft_consensus.cc:2229] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:05.661171 17995 raft_consensus.cc:2258] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:05.661602 17988 ts_tablet_manager.cc:1905] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:05.661744 17995 ts_tablet_manager.cc:1905] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:05.664227 17995 ts_tablet_manager.cc:1918] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 15:54:05.664311 17995 log.cc:1196] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/c8f8eebdfff8424598efe7ff054fe741
I20260806 15:54:05.664618 17995 ts_tablet_manager.cc:1939] T c8f8eebdfff8424598efe7ff054fe741 P 754db59a3f764a579703a95ffe8bb371: Deleting consensus metadata
I20260806 15:54:05.665298 17017 catalog_manager.cc:4841] TS 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721): tablet c8f8eebdfff8424598efe7ff054fe741 (table test-table [id=0ddd923da8ec46d6b1b3e868f9f3c33e]) successfully deleted
I20260806 15:54:05.666800 17988 ts_tablet_manager.cc:1918] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 15:54:05.666898 17988 log.cc:1196] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/c8f8eebdfff8424598efe7ff054fe741
I20260806 15:54:05.667142 17988 ts_tablet_manager.cc:1939] T c8f8eebdfff8424598efe7ff054fe741 P fe96b06192d14756aff5ae81f2b640d3: Deleting consensus metadata
I20260806 15:54:05.667812 17024 catalog_manager.cc:4841] TS fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227): tablet c8f8eebdfff8424598efe7ff054fe741 (table test-table [id=0ddd923da8ec46d6b1b3e868f9f3c33e]) successfully deleted
Aug 06 15:54:05 dist-test-slave-rwrg krb5kdc[16968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 15:54:05 dist-test-slave-rwrg krb5kdc[16968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:05.684932 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:05.676017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.13.223.62:43745)
0806 15:54:05.676285 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:05.676292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:05.676348 (+ 56us) server_negotiation.cc:407] Connection header received
0806 15:54:05.676455 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:05.676457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:05.676572 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:05.676735 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:05.677405 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:05.678016 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:05.679109 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:05.679356 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:05.679431 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:05.682117 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:05.682120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:05.682122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:05.682144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:05.683487 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:05.683858 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:05.683861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:05.683862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:05.683937 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:05.684170 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:05.684172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:05.684173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:05.684335 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:05.684433 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:05.684711 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 15:54:05.684811 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
I20260806 15:54:05.687613 17039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:05.688072 17039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:05.692724 17253 tablet_service.cc:1467] Processing CreateTablet for tablet 5a660c24b4c54f3599860f3c0afd50c7 (DEFAULT_TABLE table=test-table [id=6c91d8c4a86346a4b6bbf772e347e607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:05.692736 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 5a660c24b4c54f3599860f3c0afd50c7 (DEFAULT_TABLE table=test-table [id=6c91d8c4a86346a4b6bbf772e347e607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:05.693037 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a660c24b4c54f3599860f3c0afd50c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:05.693037 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a660c24b4c54f3599860f3c0afd50c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:05.692952 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 5a660c24b4c54f3599860f3c0afd50c7 (DEFAULT_TABLE table=test-table [id=6c91d8c4a86346a4b6bbf772e347e607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:05.693281 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a660c24b4c54f3599860f3c0afd50c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:05.695513 18011 tablet_bootstrap.cc:492] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Bootstrap starting.
I20260806 15:54:05.695799 18007 tablet_bootstrap.cc:492] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Bootstrap starting.
I20260806 15:54:05.695814 18012 tablet_bootstrap.cc:492] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Bootstrap starting.
I20260806 15:54:05.696452 18011 tablet_bootstrap.cc:654] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:05.696676 18007 tablet_bootstrap.cc:654] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:05.696722 18012 tablet_bootstrap.cc:654] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:05.697904 18011 tablet_bootstrap.cc:492] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: No bootstrap required, opened a new log
I20260806 15:54:05.697993 18011 ts_tablet_manager.cc:1397] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:05.698021 18007 tablet_bootstrap.cc:492] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: No bootstrap required, opened a new log
I20260806 15:54:05.698100 18007 ts_tablet_manager.cc:1397] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:05.698446 18007 raft_consensus.cc:348] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.698446 18011 raft_consensus.cc:348] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.698549 18011 raft_consensus.cc:374] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:05.698549 18007 raft_consensus.cc:374] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:05.698583 18011 raft_consensus.cc:729] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe96b06192d14756aff5ae81f2b640d3, State: Initialized, Role: FOLLOWER
I20260806 15:54:05.698597 18007 raft_consensus.cc:729] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Initialized, Role: FOLLOWER
I20260806 15:54:05.698588 18012 tablet_bootstrap.cc:492] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: No bootstrap required, opened a new log
I20260806 15:54:05.698743 18012 ts_tablet_manager.cc:1397] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:05.698716 18007 consensus_queue.cc:260] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.698716 18011 consensus_queue.cc:260] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.698989 18007 ts_tablet_manager.cc:1428] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:05.699219 18012 raft_consensus.cc:348] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.699321 18012 raft_consensus.cc:374] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:05.699352 18012 raft_consensus.cc:729] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc67b3f5d00848f391ba6f784174d284, State: Initialized, Role: FOLLOWER
I20260806 15:54:05.699358 18011 ts_tablet_manager.cc:1428] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 15:54:05.699498 18012 consensus_queue.cc:260] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.699929 18012 ts_tablet_manager.cc:1428] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:05.754621 17947 raft_consensus.cc:482] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:05.754758 17947 raft_consensus.cc:504] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.755177 17947 leader_election.cc:288] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:05.755667 17277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a660c24b4c54f3599860f3c0afd50c7" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3" is_pre_election: true
I20260806 15:54:05.755692 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a660c24b4c54f3599860f3c0afd50c7" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284" is_pre_election: true
I20260806 15:54:05.755805 17277 raft_consensus.cc:2454] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:05.755867 17756 raft_consensus.cc:2454] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:05.756212 17395 leader_election.cc:302] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, dc67b3f5d00848f391ba6f784174d284; no voters:
I20260806 15:54:05.756474 17947 raft_consensus.cc:2781] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:05.756564 17947 raft_consensus.cc:482] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:05.756634 17947 raft_consensus.cc:3037] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:05.757848 17947 raft_consensus.cc:504] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.758183 17947 leader_election.cc:288] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Requested vote from peers fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:05.758608 17277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a660c24b4c54f3599860f3c0afd50c7" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3"
I20260806 15:54:05.758695 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a660c24b4c54f3599860f3c0afd50c7" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284"
I20260806 15:54:05.758728 17277 raft_consensus.cc:3037] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:05.758822 17756 raft_consensus.cc:3037] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:05.759712 17756 raft_consensus.cc:2454] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:05.759903 17277 raft_consensus.cc:2454] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:05.760020 17395 leader_election.cc:302] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, dc67b3f5d00848f391ba6f784174d284; no voters:
I20260806 15:54:05.760185 17947 raft_consensus.cc:2781] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:05.760288 17947 raft_consensus.cc:686] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Becoming Leader. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Running, Role: LEADER
I20260806 15:54:05.760433 17947 consensus_queue.cc:237] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.762287 17038 catalog_manager.cc:5495] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754db59a3f764a579703a95ffe8bb371 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "754db59a3f764a579703a95ffe8bb371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:05.785995 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:05.782380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.13.223.2:44721)
0806 15:54:05.782783 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:05.782790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:05.782800 (+ 10us) server_negotiation.cc:407] Connection header received
0806 15:54:05.782838 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:05.782840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:05.782880 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:05.783119 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:05.783396 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:05.783930 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:05.784897 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:05.785128 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:05.785247 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:05.785525 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:05.785599 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:05.785684 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:05.785784 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 15:54:05.785838 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260806 15:54:05.792560 17277 raft_consensus.cc:1261] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:05.792560 17756 raft_consensus.cc:1261] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:05.792953 17947 consensus_queue.cc:1035] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:05.793154 17975 consensus_queue.cc:1035] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:05.821871 17040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:05.822016 17040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:05.824065 17040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Sending DeleteTablet for 3 replicas of tablet 5a660c24b4c54f3599860f3c0afd50c7
I20260806 15:54:05.824638 17253 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a660c24b4c54f3599860f3c0afd50c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:05 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:57434
I20260806 15:54:05.824680 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a660c24b4c54f3599860f3c0afd50c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:05 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:43632
I20260806 15:54:05.824682 17479 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a660c24b4c54f3599860f3c0afd50c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:05 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:54538
I20260806 15:54:05.824868 17988 tablet_replica.cc:330] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: stopping tablet replica
I20260806 15:54:05.824870 17995 tablet_replica.cc:330] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: stopping tablet replica
I20260806 15:54:05.824998 17988 raft_consensus.cc:2229] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:05.825007 17995 raft_consensus.cc:2229] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:05.825172 17988 raft_consensus.cc:2258] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:05.825242 17995 raft_consensus.cc:2258] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:05.825426 17998 tablet_replica.cc:330] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: stopping tablet replica
I20260806 15:54:05.825546 17998 raft_consensus.cc:2229] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:05.825675 17998 raft_consensus.cc:2258] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:05.825909 17988 ts_tablet_manager.cc:1905] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:05.826503 17998 ts_tablet_manager.cc:1905] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:05.826900 17995 ts_tablet_manager.cc:1905] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:05.826839 17040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:05.827278 17040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:05.828857 17988 ts_tablet_manager.cc:1918] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:05.828948 17988 log.cc:1196] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/5a660c24b4c54f3599860f3c0afd50c7
I20260806 15:54:05.829206 17998 ts_tablet_manager.cc:1918] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:05.829262 17988 ts_tablet_manager.cc:1939] T 5a660c24b4c54f3599860f3c0afd50c7 P fe96b06192d14756aff5ae81f2b640d3: Deleting consensus metadata
I20260806 15:54:05.829306 17998 log.cc:1196] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/5a660c24b4c54f3599860f3c0afd50c7
I20260806 15:54:05.829393 17995 ts_tablet_manager.cc:1918] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:05.829499 17995 log.cc:1196] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/5a660c24b4c54f3599860f3c0afd50c7
I20260806 15:54:05.829578 17998 ts_tablet_manager.cc:1939] T 5a660c24b4c54f3599860f3c0afd50c7 P dc67b3f5d00848f391ba6f784174d284: Deleting consensus metadata
I20260806 15:54:05.829773 17995 ts_tablet_manager.cc:1939] T 5a660c24b4c54f3599860f3c0afd50c7 P 754db59a3f764a579703a95ffe8bb371: Deleting consensus metadata
I20260806 15:54:05.830053 17024 catalog_manager.cc:4841] TS fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227): tablet 5a660c24b4c54f3599860f3c0afd50c7 (table test-table [id=6c91d8c4a86346a4b6bbf772e347e607]) successfully deleted
I20260806 15:54:05.830240 17024 catalog_manager.cc:4841] TS dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829): tablet 5a660c24b4c54f3599860f3c0afd50c7 (table test-table [id=6c91d8c4a86346a4b6bbf772e347e607]) successfully deleted
I20260806 15:54:05.830513 17017 catalog_manager.cc:4841] TS 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721): tablet 5a660c24b4c54f3599860f3c0afd50c7 (table test-table [id=6c91d8c4a86346a4b6bbf772e347e607]) successfully deleted
I20260806 15:54:05.834270 17253 tablet_service.cc:1467] Processing CreateTablet for tablet d0f1474828a54ac59e79d434fd0ce07d (DEFAULT_TABLE table=test-table [id=7811de98d935437ba62c16c8eca27118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:05.834287 17479 tablet_service.cc:1467] Processing CreateTablet for tablet d0f1474828a54ac59e79d434fd0ce07d (DEFAULT_TABLE table=test-table [id=7811de98d935437ba62c16c8eca27118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:05.834580 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f1474828a54ac59e79d434fd0ce07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:05.834580 17717 tablet_service.cc:1467] Processing CreateTablet for tablet d0f1474828a54ac59e79d434fd0ce07d (DEFAULT_TABLE table=test-table [id=7811de98d935437ba62c16c8eca27118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:05.834646 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f1474828a54ac59e79d434fd0ce07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:05.834829 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f1474828a54ac59e79d434fd0ce07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:05.836606 18007 tablet_bootstrap.cc:492] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Bootstrap starting.
I20260806 15:54:05.837061 18011 tablet_bootstrap.cc:492] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3: Bootstrap starting.
I20260806 15:54:05.837183 18012 tablet_bootstrap.cc:492] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Bootstrap starting.
I20260806 15:54:05.837527 18007 tablet_bootstrap.cc:654] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:05.837917 18011 tablet_bootstrap.cc:654] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:05.838008 18012 tablet_bootstrap.cc:654] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:05.838716 18007 tablet_bootstrap.cc:492] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: No bootstrap required, opened a new log
I20260806 15:54:05.838816 18007 ts_tablet_manager.cc:1397] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:05.839056 18011 tablet_bootstrap.cc:492] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3: No bootstrap required, opened a new log
I20260806 15:54:05.839147 18011 ts_tablet_manager.cc:1397] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:05.839264 18007 raft_consensus.cc:348] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.839377 18007 raft_consensus.cc:374] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:05.839411 18007 raft_consensus.cc:729] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Initialized, Role: FOLLOWER
I20260806 15:54:05.839542 18007 consensus_queue.cc:260] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.839587 18011 raft_consensus.cc:348] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.839713 18011 raft_consensus.cc:374] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:05.839771 18011 raft_consensus.cc:729] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe96b06192d14756aff5ae81f2b640d3, State: Initialized, Role: FOLLOWER
I20260806 15:54:05.839850 18007 ts_tablet_manager.cc:1428] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:05.839903 18011 consensus_queue.cc:260] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.840181 18011 ts_tablet_manager.cc:1428] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:05.841221 18012 tablet_bootstrap.cc:492] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: No bootstrap required, opened a new log
I20260806 15:54:05.841351 18012 ts_tablet_manager.cc:1397] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 15:54:05.841912 18012 raft_consensus.cc:348] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.842020 18012 raft_consensus.cc:374] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:05.842057 18012 raft_consensus.cc:729] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc67b3f5d00848f391ba6f784174d284, State: Initialized, Role: FOLLOWER
I20260806 15:54:05.842190 18012 consensus_queue.cc:260] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.842456 18012 ts_tablet_manager.cc:1428] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:54:05.917166 17980 raft_consensus.cc:482] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:05.917327 17980 raft_consensus.cc:504] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.917850 17980 leader_election.cc:288] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:05.918279 17277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f1474828a54ac59e79d434fd0ce07d" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3" is_pre_election: true
I20260806 15:54:05.918334 17754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f1474828a54ac59e79d434fd0ce07d" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284" is_pre_election: true
I20260806 15:54:05.918426 17277 raft_consensus.cc:2454] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:05.918524 17754 raft_consensus.cc:2454] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 0.
I20260806 15:54:05.918736 17395 leader_election.cc:302] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, fe96b06192d14756aff5ae81f2b640d3; no voters:
I20260806 15:54:05.918903 17980 raft_consensus.cc:2781] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:05.918963 17980 raft_consensus.cc:482] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:05.919013 17980 raft_consensus.cc:3037] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:05.920248 17980 raft_consensus.cc:504] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.920665 17980 leader_election.cc:288] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Requested vote from peers fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227), dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829)
I20260806 15:54:05.921070 17277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f1474828a54ac59e79d434fd0ce07d" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe96b06192d14756aff5ae81f2b640d3"
I20260806 15:54:05.921070 17754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f1474828a54ac59e79d434fd0ce07d" candidate_uuid: "754db59a3f764a579703a95ffe8bb371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc67b3f5d00848f391ba6f784174d284"
I20260806 15:54:05.921193 17277 raft_consensus.cc:3037] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:05.921193 17754 raft_consensus.cc:3037] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:05.922408 17277 raft_consensus.cc:2454] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:05.922540 17754 raft_consensus.cc:2454] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754db59a3f764a579703a95ffe8bb371 in term 1.
I20260806 15:54:05.922757 17395 leader_election.cc:302] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754db59a3f764a579703a95ffe8bb371, fe96b06192d14756aff5ae81f2b640d3; no voters:
I20260806 15:54:05.922961 17980 raft_consensus.cc:2781] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:05.923054 17980 raft_consensus.cc:686] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Becoming Leader. State: Replica: 754db59a3f764a579703a95ffe8bb371, State: Running, Role: LEADER
I20260806 15:54:05.923213 17980 consensus_queue.cc:237] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } }
I20260806 15:54:05.924950 17039 catalog_manager.cc:5495] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754db59a3f764a579703a95ffe8bb371 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "754db59a3f764a579703a95ffe8bb371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754db59a3f764a579703a95ffe8bb371" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:05.951149 17756 raft_consensus.cc:1261] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:05.951207 17277 raft_consensus.cc:1261] T d0f1474828a54ac59e79d434fd0ce07d P fe96b06192d14756aff5ae81f2b640d3 [term 1 FOLLOWER]: Refusing update from remote peer 754db59a3f764a579703a95ffe8bb371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:05.951504 17947 consensus_queue.cc:1035] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc67b3f5d00848f391ba6f784174d284" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:05.951969 17947 consensus_queue.cc:1035] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe96b06192d14756aff5ae81f2b640d3" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 15:54:05.965782 17958 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 15:54:07.516630 17039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:07.516779 17039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:07.519203 17039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6413ca40e3420abdb3f3da448f36d3: Sending DeleteTablet for 3 replicas of tablet d0f1474828a54ac59e79d434fd0ce07d
I20260806 15:54:07.519809 17479 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f1474828a54ac59e79d434fd0ce07d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:07 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:54538
I20260806 15:54:07.519896 17253 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f1474828a54ac59e79d434fd0ce07d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:07 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:57434
I20260806 15:54:07.519902 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f1474828a54ac59e79d434fd0ce07d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:07 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:43632
I20260806 15:54:07.520176 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17149
I20260806 15:54:07.520298 18084 tablet_replica.cc:330] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: stopping tablet replica
I20260806 15:54:07.521099 18086 tablet_replica.cc:330] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: stopping tablet replica
I20260806 15:54:07.521106 18084 raft_consensus.cc:2229] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:07.521238 18086 raft_consensus.cc:2229] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:07.521253 18084 raft_consensus.cc:2258] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:07.521412 18086 raft_consensus.cc:2258] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:07.523547 18086 ts_tablet_manager.cc:1905] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:07.523608 18084 ts_tablet_manager.cc:1905] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:07.526753 18086 ts_tablet_manager.cc:1918] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 15:54:07.526825 18086 log.cc:1196] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/d0f1474828a54ac59e79d434fd0ce07d
I20260806 15:54:07.527113 18086 ts_tablet_manager.cc:1939] T d0f1474828a54ac59e79d434fd0ce07d P 754db59a3f764a579703a95ffe8bb371: Deleting consensus metadata
I20260806 15:54:07.527408 18084 ts_tablet_manager.cc:1918] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 15:54:07.527500 18084 log.cc:1196] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/d0f1474828a54ac59e79d434fd0ce07d
I20260806 15:54:07.527889 18084 ts_tablet_manager.cc:1939] T d0f1474828a54ac59e79d434fd0ce07d P dc67b3f5d00848f391ba6f784174d284: Deleting consensus metadata
I20260806 15:54:07.530838 17024 catalog_manager.cc:4841] TS dc67b3f5d00848f391ba6f784174d284 (127.13.223.3:40829): tablet d0f1474828a54ac59e79d434fd0ce07d (table test-table [id=7811de98d935437ba62c16c8eca27118]) successfully deleted
I20260806 15:54:07.531116 17017 catalog_manager.cc:4841] TS 754db59a3f764a579703a95ffe8bb371 (127.13.223.2:44721): tablet d0f1474828a54ac59e79d434fd0ce07d (table test-table [id=7811de98d935437ba62c16c8eca27118]) successfully deleted
I20260806 15:54:07.540133 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17360
W20260806 15:54:07.540096 17024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:42227 (error 108)
I20260806 15:54:07.547322 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:07.544848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42227 (local address 127.0.0.1:54150)
0806 15:54:07.547070 (+ 2222us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:07.547176 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2106,"thread_start_us":939,"threads_started":1}
W20260806 15:54:07.547526 17024 catalog_manager.cc:4584] TS fe96b06192d14756aff5ae81f2b640d3 (127.13.223.1:42227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0f1474828a54ac59e79d434fd0ce07d: Network error: Client connection negotiation failed: client connection to 127.13.223.1:42227: connect: Connection refused (error 111)
I20260806 15:54:07.554977 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17599
I20260806 15:54:07.566452 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16990
2026-08-06T15:54:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6499 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:07 dist-test-slave-rwrg krb5kdc[18098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:07 dist-test-slave-rwrg krb5kdc[18098](info): set up 2 sockets
Aug 06 15:54:07 dist-test-slave-rwrg krb5kdc[18098](info): commencing operation
krb5kdc: starting...
W20260806 15:54:08.877452 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:08 dist-test-slave-rwrg krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:09.009895 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:41437
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:41437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:09.124701 18115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:09.125016 18115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:09.125089 18115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:09.129065 18115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:09.129158 18115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:09.129196 18115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:09.129276 18115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:09.129314 18115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:09.134572 18115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:41437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:09.135685 18115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:09.136749 18115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:09.145080 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.145121 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.145150 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:09.145481 18115 server_base.cc:1034] running on GCE node
I20260806 15:54:09.146034 18115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:09.147114 18115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:09.148278 18115 hybrid_clock.cc:648] HybridClock initialized: now 1786031649148265 us; error 26 us; skew 500 ppm
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:09.151120 18115 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:09.152036 18115 webserver.cc:460] Webserver started at http://127.13.223.62:39797/ using document root <none> and password file <none>
I20260806 15:54:09.152695 18115 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:09.152768 18115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:09.153074 18115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:09.155167 18115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "484ae0afc6af4033a808d4af72a259c0"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "d293ea17ed3121e3c6d79af5ad0ceb5e"
server_key_iv: "287ce71e172cb9cd0ea58f98ab887801"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.155746 18115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "484ae0afc6af4033a808d4af72a259c0"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "d293ea17ed3121e3c6d79af5ad0ceb5e"
server_key_iv: "287ce71e172cb9cd0ea58f98ab887801"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.159634 18115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:09.162187 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.163467 18115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:09.163638 18115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "484ae0afc6af4033a808d4af72a259c0"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "d293ea17ed3121e3c6d79af5ad0ceb5e"
server_key_iv: "287ce71e172cb9cd0ea58f98ab887801"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.163761 18115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:09.182740 18115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:09.185824 18115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:09.186075 18115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:09.193502 18115 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:41437
I20260806 15:54:09.193531 18229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:41437 every 8 connection(s)
I20260806 15:54:09.194461 18115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:09.196892 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 18115
I20260806 15:54:09.197010 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:09.197311 14204 external_mini_cluster.cc:1428] Setting key f8b9c03dc71b0bc9ecfdb0df8726c174
I20260806 15:54:09.197455 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:09.203802 18230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0: Bootstrap starting.
I20260806 15:54:09.206202 18230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0: Neither blocks nor log segments found. Creating new log.
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:09.207026 18230 log.cc:824] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:09.209028 18230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0: No bootstrap required, opened a new log
I20260806 15:54:09.211807 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.198792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.13.223.62:41437)
0806 15:54:09.199254 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:09.199268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:09.199306 (+ 38us) server_negotiation.cc:407] Connection header received
0806 15:54:09.200197 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:09.200216 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:09.200589 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:09.200923 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:09.201755 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.202744 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.203615 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.204184 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.204355 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.207161 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:09.207166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:09.207180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:09.207207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:09.209003 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.209496 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.209504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.209514 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.209628 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.209913 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.209918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.209920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.210312 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:09.210517 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:09.210819 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 15:54:09.211113 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260806 15:54:09.212852 18230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ae0afc6af4033a808d4af72a259c0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 41437 } }
I20260806 15:54:09.213166 18230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:09.213229 18230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484ae0afc6af4033a808d4af72a259c0, State: Initialized, Role: FOLLOWER
I20260806 15:54:09.213692 18230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ae0afc6af4033a808d4af72a259c0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 41437 } }
I20260806 15:54:09.213855 18230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:09.213920 18230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:09.214015 18230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:09.215011 18230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ae0afc6af4033a808d4af72a259c0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 41437 } }
I20260806 15:54:09.215360 18230 leader_election.cc:302] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484ae0afc6af4033a808d4af72a259c0; no voters:
I20260806 15:54:09.215665 18230 leader_election.cc:288] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:09.215817 18235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:09.216068 18235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [term 1 LEADER]: Becoming Leader. State: Replica: 484ae0afc6af4033a808d4af72a259c0, State: Running, Role: LEADER
I20260806 15:54:09.216382 18235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ae0afc6af4033a808d4af72a259c0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 41437 } }
I20260806 15:54:09.216904 18230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:09.217980 18236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484ae0afc6af4033a808d4af72a259c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ae0afc6af4033a808d4af72a259c0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 41437 } } }
I20260806 15:54:09.218138 18236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:09.218472 18238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484ae0afc6af4033a808d4af72a259c0. Latest consensus state: current_term: 1 leader_uuid: "484ae0afc6af4033a808d4af72a259c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484ae0afc6af4033a808d4af72a259c0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 41437 } } }
I20260806 15:54:09.218562 18238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:09.219410 18247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:09.222126 18247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:09.227926 18247 catalog_manager.cc:1347] Generated new cluster ID: 8259f5ee298042db819afeb87f183d00
I20260806 15:54:09.228004 18247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:09.234251 18247 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:09.235147 18247 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:09.239823 18247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484ae0afc6af4033a808d4af72a259c0: Generated new TSK 0
I20260806 15:54:09.240468 18247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:09.277071 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:41437
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:09.390466 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:09.390736 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:09.390779 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:09.394855 18263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:09.394910 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:09.395015 18263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:09.400259 18263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:41437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:09.401360 18263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:09.402402 18263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:09.411954 18272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.411988 18271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.411988 18275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:09.412349 18263 server_base.cc:1034] running on GCE node
I20260806 15:54:09.412770 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:09.413434 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:09.414618 18263 hybrid_clock.cc:648] HybridClock initialized: now 1786031649414584 us; error 49 us; skew 500 ppm
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:09.417382 18263 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:09.418361 18263 webserver.cc:460] Webserver started at http://127.13.223.1:43749/ using document root <none> and password file <none>
I20260806 15:54:09.418968 18263 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:09.419059 18263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:09.419337 18263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:09.421391 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "38912c342e5c4953977a09d712ea5edf"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "0db8444b6d03c5ea538a63731a180f65"
server_key_iv: "5f698f04e6325d8670a3aee32feed6da"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.421929 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "38912c342e5c4953977a09d712ea5edf"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "0db8444b6d03c5ea538a63731a180f65"
server_key_iv: "5f698f04e6325d8670a3aee32feed6da"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.425493 18263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 15:54:09.427889 18286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.429149 18263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:54:09.429315 18263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "38912c342e5c4953977a09d712ea5edf"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "0db8444b6d03c5ea538a63731a180f65"
server_key_iv: "5f698f04e6325d8670a3aee32feed6da"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.429450 18263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:09.458962 18263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:09.462278 18263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:09.462479 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:09.463073 18263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:09.463899 18263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:09.463981 18263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.464072 18263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:09.464121 18263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.473626 18263 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:38669
I20260806 15:54:09.473651 18471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:38669 every 8 connection(s)
I20260806 15:54:09.474522 18263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:09.484287 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 18263
I20260806 15:54:09.484431 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:09.484764 14204 external_mini_cluster.cc:1428] Setting key 27926e614729efc079a049593032254f
I20260806 15:54:09.488127 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.476381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:33327 (local address 127.13.223.62:41437)
0806 15:54:09.476558 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:09.476563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:09.477651 (+ 1088us) server_negotiation.cc:407] Connection header received
0806 15:54:09.478465 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:09.478468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:09.478514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:09.478587 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:09.479712 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.480224 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.481385 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.481629 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.481708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.484088 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:09.484094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:09.484097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:09.484126 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:09.485800 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.486498 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.486504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.486506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.486620 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.487030 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.487032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.487033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.487200 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:09.487274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:09.487908 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 15:54:09.488011 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:54:09.489107 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.476842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:41437 (local address 127.13.223.1:33327)
0806 15:54:09.477500 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:09.477540 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:09.478290 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:09.478769 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:09.478775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:09.479115 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:09.479579 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:09.479592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.480330 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.480332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:09.481278 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:09.481288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.481771 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.481774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:09.481909 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.482446 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:09.482462 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:09.483914 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:09.485921 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:09.485926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:09.485937 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:09.486308 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:09.486730 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:09.486734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:09.486736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:09.486933 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:09.487305 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:09.487315 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:09.487635 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 15:54:09.487866 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 15:54:09.488217 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":114,"threads_started":1}
I20260806 15:54:09.490311 18472 heartbeater.cc:347] Connected to a master server at 127.13.223.62:41437
I20260806 15:54:09.490562 18472 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:09.491149 18472 heartbeater.cc:511] Master 127.13.223.62:41437 requested a full tablet report, sending...
I20260806 15:54:09.492786 18165 ts_manager.cc:194] Registered new tserver with Master: 38912c342e5c4953977a09d712ea5edf (127.13.223.1:38669)
I20260806 15:54:09.493873 18165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:33327
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:09.515066 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:41437
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:09.629189 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:09.629469 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:09.629513 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:09.633466 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:09.633520 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:09.633581 18477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:09.638710 18477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:41437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:09.639717 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:09.640763 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:09.650401 18490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.650395 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.650477 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:09.650769 18477 server_base.cc:1034] running on GCE node
I20260806 15:54:09.651216 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:09.651892 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:09.653052 18477 hybrid_clock.cc:648] HybridClock initialized: now 1786031649653039 us; error 32 us; skew 500 ppm
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:09.655911 18477 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:09.656901 18477 webserver.cc:460] Webserver started at http://127.13.223.2:45247/ using document root <none> and password file <none>
I20260806 15:54:09.657429 18477 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:09.657487 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:09.657651 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:09.659667 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "c369d8829a064850935a63fc0225af7d"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "4a92f78d2c03a410795f3b9a7d96290f"
server_key_iv: "b0db2f498b66cfe422d95de62e1706f9"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.660197 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "c369d8829a064850935a63fc0225af7d"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "4a92f78d2c03a410795f3b9a7d96290f"
server_key_iv: "b0db2f498b66cfe422d95de62e1706f9"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.664196 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:09.666785 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.668164 18477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:09.668291 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "c369d8829a064850935a63fc0225af7d"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "4a92f78d2c03a410795f3b9a7d96290f"
server_key_iv: "b0db2f498b66cfe422d95de62e1706f9"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.668381 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:09.697379 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:09.700937 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:09.701092 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:09.701658 18477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:09.702540 18477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:09.702597 18477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.702641 18477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:09.702656 18477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.718014 18477 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:39915
I20260806 15:54:09.718178 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:39915 every 8 connection(s)
I20260806 15:54:09.719347 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:09.722746 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 18477
I20260806 15:54:09.722864 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:09.723165 14204 external_mini_cluster.cc:1428] Setting key 60b8dda706298e3a537511b057bc0325
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:09.737852 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.721585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:48117 (local address 127.13.223.62:41437)
0806 15:54:09.721735 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:09.721740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:09.723344 (+ 1604us) server_negotiation.cc:407] Connection header received
0806 15:54:09.724665 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:09.724670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:09.724731 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:09.724810 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:09.726971 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.727715 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.729642 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.729889 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.730016 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.733175 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:09.733179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:09.733182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:09.733241 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:09.735154 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.735744 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.735751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.735754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.735859 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.736268 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.736274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.736279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.736593 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:09.736710 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:09.737495 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 15:54:09.737638 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:54:09.738911 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.722207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:41437 (local address 127.13.223.2:48117)
0806 15:54:09.723139 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:09.723203 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:09.724337 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:09.725267 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:09.725279 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:09.725891 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:09.726784 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:09.726810 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.727896 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.727903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:09.729480 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:09.729495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.730065 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.730069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:09.730263 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.730971 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:09.730989 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:09.732982 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:09.735289 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:09.735294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:09.735306 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:09.735609 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:09.735964 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:09.735966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:09.735967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:09.736119 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:09.736742 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:09.736751 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:09.737123 (+ 372us) client_negotiation.cc:769] Sending connection context
0806 15:54:09.737417 (+ 294us) client_negotiation.cc:240] Negotiation successful
0806 15:54:09.737824 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":121,"threads_started":1}
I20260806 15:54:09.740259 18678 heartbeater.cc:347] Connected to a master server at 127.13.223.62:41437
I20260806 15:54:09.740552 18678 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:09.741257 18678 heartbeater.cc:511] Master 127.13.223.62:41437 requested a full tablet report, sending...
I20260806 15:54:09.742398 18165 ts_manager.cc:194] Registered new tserver with Master: c369d8829a064850935a63fc0225af7d (127.13.223.2:39915)
I20260806 15:54:09.743314 18165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:48117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:09.763734 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:41437
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:09.877187 18685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:09.877430 18685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:09.877471 18685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:09.881577 18685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:09.881626 18685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:09.881713 18685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:09.886797 18685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:41437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:09.887895 18685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:09.888952 18685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:09.899277 18703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:09.899287 18699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:09.899358 18685 server_base.cc:1034] running on GCE node
W20260806 15:54:09.899287 18698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:09.899907 18685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:09.900638 18685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:09.901808 18685 hybrid_clock.cc:648] HybridClock initialized: now 1786031649901791 us; error 31 us; skew 500 ppm
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:09.904603 18685 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:09.905556 18685 webserver.cc:460] Webserver started at http://127.13.223.3:46717/ using document root <none> and password file <none>
I20260806 15:54:09.906076 18685 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:09.906133 18685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:09.906304 18685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:09.908303 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "ba259e89584c41df9613b208cf778a01"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "7f8bb6622a3c413c73b8e28051783684"
server_key_iv: "c06e7dffbb4e4e991ac666e626ca9084"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.908892 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "ba259e89584c41df9613b208cf778a01"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "7f8bb6622a3c413c73b8e28051783684"
server_key_iv: "c06e7dffbb4e4e991ac666e626ca9084"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.912686 18685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:09.915045 18716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.916350 18685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:09.916564 18685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "ba259e89584c41df9613b208cf778a01"
format_stamp: "Formatted at 2026-08-06 15:54:09 on dist-test-slave-rwrg"
server_key: "7f8bb6622a3c413c73b8e28051783684"
server_key_iv: "c06e7dffbb4e4e991ac666e626ca9084"
server_key_version: "encryptionkey@0"
I20260806 15:54:09.916699 18685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:09.927510 18685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:09.930899 18685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:09.931056 18685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:09.931591 18685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:09.932394 18685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:09.932440 18685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.932479 18685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:09.932559 18685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:09.943920 18685 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:36561
I20260806 15:54:09.943948 18878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:36561 every 8 connection(s)
I20260806 15:54:09.944917 18685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:09.951041 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 18685
I20260806 15:54:09.951151 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:09.951475 14204 external_mini_cluster.cc:1428] Setting key 55a19c4800166b165992c8aa7b521cae
Aug 06 15:54:09 dist-test-slave-rwrg krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031649, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:09.958818 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.946719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:40697 (local address 127.13.223.62:41437)
0806 15:54:09.946864 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:09.946867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:09.947943 (+ 1076us) server_negotiation.cc:407] Connection header received
0806 15:54:09.948896 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:09.948899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:09.948949 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:09.949027 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:09.950190 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.950729 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.952178 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.952475 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.952636 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.955344 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:09.955347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:09.955349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:09.955372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:09.956788 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.957321 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.957325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.957327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.957410 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:09.957762 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:09.957766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:09.957768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:09.957949 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:09.958067 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:09.958544 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 15:54:09.958659 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:54:09.959635 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:09.947133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:41437 (local address 127.13.223.3:40697)
0806 15:54:09.947805 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:09.947846 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:09.948702 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:09.949201 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:09.949208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:09.949579 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:09.950049 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:09.950062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.950881 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.950884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:09.952047 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:09.952057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:09.952674 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:09.952679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:09.952846 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:09.953547 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:09.953570 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:09.955160 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:09.956914 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:09.956919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:09.956934 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:09.957211 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:09.957505 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:09.957509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:09.957511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:09.957660 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:09.958060 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:09.958066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:09.958312 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 15:54:09.958503 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 15:54:09.958830 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":120,"threads_started":1}
I20260806 15:54:09.960777 18879 heartbeater.cc:347] Connected to a master server at 127.13.223.62:41437
I20260806 15:54:09.960999 18879 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:09.961445 18879 heartbeater.cc:511] Master 127.13.223.62:41437 requested a full tablet report, sending...
I20260806 15:54:09.962450 18165 ts_manager.cc:194] Registered new tserver with Master: ba259e89584c41df9613b208cf778a01 (127.13.223.3:36561)
I20260806 15:54:09.962962 18165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:40697
I20260806 15:54:09.965628 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 15:54:09.981871 18233 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:09.980481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.13.223.62:41437)
0806 15:54:09.980689 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:09.980693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:09.980847 (+ 154us) server_negotiation.cc:407] Connection header received
0806 15:54:09.980934 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:09.980936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:09.980986 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:09.981069 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:09.981759 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52788: BlockingRecv error: recv got EOF from 127.0.0.1:52788 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 15:54:09.983510 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 18263
I20260806 15:54:09.996812 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 18477
I20260806 15:54:10.011072 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 18685
I20260806 15:54:10.023926 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 18115
2026-08-06T15:54:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2456 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:10 dist-test-slave-rwrg krb5kdc[18896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:10 dist-test-slave-rwrg krb5kdc[18896](info): set up 2 sockets
Aug 06 15:54:10 dist-test-slave-rwrg krb5kdc[18896](info): commencing operation
krb5kdc: starting...
W20260806 15:54:11.295301 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:11 dist-test-slave-rwrg krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:11.425773 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:37851
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:37851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:11.539811 18917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:11.540120 18917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:11.540207 18917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:11.544216 18917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:11.544315 18917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:11.544359 18917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:11.544425 18917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:11.544471 18917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:11.549856 18917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:37851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:11.550951 18917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:11.552019 18917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:11.561312 18924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:11.561348 18923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:11.561506 18917 server_base.cc:1034] running on GCE node
W20260806 15:54:11.561314 18927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:11.562294 18917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:11.563434 18917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:11.564635 18917 hybrid_clock.cc:648] HybridClock initialized: now 1786031651564605 us; error 40 us; skew 500 ppm
Aug 06 15:54:11 dist-test-slave-rwrg krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031651, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:11.568655 18917 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:11.569720 18917 webserver.cc:460] Webserver started at http://127.13.223.62:42157/ using document root <none> and password file <none>
I20260806 15:54:11.570334 18917 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:11.570405 18917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:11.570695 18917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:11.573081 18917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "3384b17ed10544048337929548c20d7b"
format_stamp: "Formatted at 2026-08-06 15:54:11 on dist-test-slave-rwrg"
server_key: "d0d13ed77633186f5744de2b975424d9"
server_key_iv: "e084d6fcadbac9052297bda534698280"
server_key_version: "encryptionkey@0"
I20260806 15:54:11.573711 18917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "3384b17ed10544048337929548c20d7b"
format_stamp: "Formatted at 2026-08-06 15:54:11 on dist-test-slave-rwrg"
server_key: "d0d13ed77633186f5744de2b975424d9"
server_key_iv: "e084d6fcadbac9052297bda534698280"
server_key_version: "encryptionkey@0"
I20260806 15:54:11.577775 18917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:11.580363 18939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:11.581668 18917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:11.581811 18917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3384b17ed10544048337929548c20d7b"
format_stamp: "Formatted at 2026-08-06 15:54:11 on dist-test-slave-rwrg"
server_key: "d0d13ed77633186f5744de2b975424d9"
server_key_iv: "e084d6fcadbac9052297bda534698280"
server_key_version: "encryptionkey@0"
I20260806 15:54:11.581928 18917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:11.601748 18917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:11.604862 18917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:11.605093 18917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:11.612648 18917 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:37851
I20260806 15:54:11.612957 19024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:37851 every 8 connection(s)
I20260806 15:54:11.613701 18917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:11.616459 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:11.622076 19027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b: Bootstrap starting.
I20260806 15:54:11.622550 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 18917
I20260806 15:54:11.622681 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:11.622946 14204 external_mini_cluster.cc:1428] Setting key fafb14fd5c1932457d6ef401bd7e0ef3
I20260806 15:54:11.624765 19027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:11.625487 19027 log.cc:824] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:11.627300 19027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b: No bootstrap required, opened a new log
I20260806 15:54:11.630903 19027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3384b17ed10544048337929548c20d7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37851 } }
I20260806 15:54:11.631145 19027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:11.631230 19027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3384b17ed10544048337929548c20d7b, State: Initialized, Role: FOLLOWER
Aug 06 15:54:11 dist-test-slave-rwrg krb5kdc[18896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:11.631732 19027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3384b17ed10544048337929548c20d7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37851 } }
I20260806 15:54:11.631908 19027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:11.631983 19027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:11.632087 19027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:11.633212 19027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3384b17ed10544048337929548c20d7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37851 } }
I20260806 15:54:11.633584 19027 leader_election.cc:302] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3384b17ed10544048337929548c20d7b; no voters:
I20260806 15:54:11.633913 19027 leader_election.cc:288] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:11.634025 19034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:11.634262 19034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [term 1 LEADER]: Becoming Leader. State: Replica: 3384b17ed10544048337929548c20d7b, State: Running, Role: LEADER
I20260806 15:54:11.634594 19034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3384b17ed10544048337929548c20d7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37851 } }
I20260806 15:54:11.635249 19027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:11.636214 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3384b17ed10544048337929548c20d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3384b17ed10544048337929548c20d7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37851 } } }
I20260806 15:54:11.636344 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:11.636621 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3384b17ed10544048337929548c20d7b. Latest consensus state: current_term: 1 leader_uuid: "3384b17ed10544048337929548c20d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3384b17ed10544048337929548c20d7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37851 } } }
I20260806 15:54:11.636710 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:11.637283 19048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:11.638441 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:11.624182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.13.223.62:37851)
0806 15:54:11.624653 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:11.624676 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:11.624710 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:54:11.625350 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:11.625366 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:11.625745 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:11.626090 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:11.626877 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:11.627875 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:11.628836 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:11.629307 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:11.629484 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:11.632202 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:11.632210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:11.632227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:11.632258 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:11.634353 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:11.634910 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:11.634916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:11.634922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:11.635069 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:11.635423 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:11.635427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:11.635428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:11.635931 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:11.636967 (+ 1036us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:11.637162 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 15:54:11.637503 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":153,"threads_started":1}
I20260806 15:54:11.640409 19048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:11.641263 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:54:11.646044 19048 catalog_manager.cc:1347] Generated new cluster ID: 2c0a0a0b145947f08405101830bbf89a
I20260806 15:54:11.646099 19048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:11.659662 19048 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:11.660849 19048 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:11.662494 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:11.656679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.13.223.62:37851)
0806 15:54:11.656787 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:11.656793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:11.657083 (+ 290us) server_negotiation.cc:407] Connection header received
0806 15:54:11.657213 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:11.657216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:11.657271 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:11.657362 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:11.657987 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:11.658767 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:11.659902 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:11.660307 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:11.660416 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:11.661414 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:11.661419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:11.661422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:11.661989 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:11.662023 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:11.662044 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:11.662133 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:11.662267 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 15:54:11.662342 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260806 15:54:11.669092 19048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3384b17ed10544048337929548c20d7b: Generated new TSK 0
I20260806 15:54:11.669786 19048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:11.675763 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 18917
2026-08-06T15:54:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1655 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:11 dist-test-slave-rwrg krb5kdc[19079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:11 dist-test-slave-rwrg krb5kdc[19079](info): set up 2 sockets
Aug 06 15:54:11 dist-test-slave-rwrg krb5kdc[19079](info): commencing operation
krb5kdc: starting...
W20260806 15:54:12.928932 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:13.053676 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:38209
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:13.166477 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:13.166811 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:13.166913 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:13.170977 19096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:13.171082 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:13.171126 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:13.171178 19096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:13.171222 19096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:13.176637 19096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:13.177824 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:13.178921 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:13.188426 19102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:13.188414 19111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:13.188485 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:13.188753 19096 server_base.cc:1034] running on GCE node
I20260806 15:54:13.189296 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:13.190518 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:13.191715 19096 hybrid_clock.cc:648] HybridClock initialized: now 1786031653191677 us; error 52 us; skew 500 ppm
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:13.198309 19096 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:13.199344 19096 webserver.cc:460] Webserver started at http://127.13.223.62:38799/ using document root <none> and password file <none>
I20260806 15:54:13.199921 19096 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:13.200016 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:13.200280 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:13.202404 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "86bc7c0617d64d07bc8c433a4cf15156"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "b7f141a19b73c2fea24d52dac581ca18"
server_key_iv: "363e94d89ef48fddde7cde706eba52c5"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.202971 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "86bc7c0617d64d07bc8c433a4cf15156"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "b7f141a19b73c2fea24d52dac581ca18"
server_key_iv: "363e94d89ef48fddde7cde706eba52c5"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.206818 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:13.209620 19122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.211050 19096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:13.211220 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "86bc7c0617d64d07bc8c433a4cf15156"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "b7f141a19b73c2fea24d52dac581ca18"
server_key_iv: "363e94d89ef48fddde7cde706eba52c5"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.211342 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:13.221966 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:13.225476 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:13.225698 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:13.233834 19096 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:38209
I20260806 15:54:13.234133 19201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:38209 every 8 connection(s)
I20260806 15:54:13.234927 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:13.237877 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:13.240585 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 19096
I20260806 15:54:13.240763 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:13.241091 14204 external_mini_cluster.cc:1428] Setting key 9ddb6b8bb159e8d4886778f0efabe032
I20260806 15:54:13.243824 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156: Bootstrap starting.
I20260806 15:54:13.246065 19204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:13.246811 19204 log.cc:824] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:13.248694 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156: No bootstrap required, opened a new log
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:13.252449 19204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc7c0617d64d07bc8c433a4cf15156" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38209 } }
I20260806 15:54:13.252777 19204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:13.252853 19204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc7c0617d64d07bc8c433a4cf15156, State: Initialized, Role: FOLLOWER
I20260806 15:54:13.253320 19204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc7c0617d64d07bc8c433a4cf15156" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38209 } }
I20260806 15:54:13.253443 19204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:13.253525 19204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:13.253674 19204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:13.254516 19204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc7c0617d64d07bc8c433a4cf15156" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38209 } }
I20260806 15:54:13.254858 19204 leader_election.cc:302] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86bc7c0617d64d07bc8c433a4cf15156; no voters:
I20260806 15:54:13.255201 19204 leader_election.cc:288] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:13.255267 19214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:13.255463 19214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [term 1 LEADER]: Becoming Leader. State: Replica: 86bc7c0617d64d07bc8c433a4cf15156, State: Running, Role: LEADER
I20260806 15:54:13.255807 19214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc7c0617d64d07bc8c433a4cf15156" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38209 } }
I20260806 15:54:13.256484 19204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:13.257063 19214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86bc7c0617d64d07bc8c433a4cf15156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc7c0617d64d07bc8c433a4cf15156" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38209 } } }
I20260806 15:54:13.257252 19214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:13.257337 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:13.242682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.13.223.62:38209)
0806 15:54:13.243139 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:13.243155 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:13.243192 (+ 37us) server_negotiation.cc:407] Connection header received
0806 15:54:13.243913 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:13.243940 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:13.244320 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:13.244723 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:13.245620 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.246772 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.247750 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.248263 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.248460 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:13.251405 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:13.251410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:13.251423 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:13.251453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:13.253708 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:13.254215 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:13.254221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:13.254227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:13.254358 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:13.254726 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:13.254729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:13.254730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:13.255268 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:13.255541 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:13.255873 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 15:54:13.256309 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260806 15:54:13.257799 19223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:13.257513 19214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bc7c0617d64d07bc8c433a4cf15156. Latest consensus state: current_term: 1 leader_uuid: "86bc7c0617d64d07bc8c433a4cf15156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc7c0617d64d07bc8c433a4cf15156" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38209 } } }
I20260806 15:54:13.258770 19214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:13.261509 19223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:13.266673 19223 catalog_manager.cc:1347] Generated new cluster ID: a7703215689f493abc72292388b3e43f
I20260806 15:54:13.266742 19223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:13.279817 19223 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:13.280807 19223 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:13.289062 19223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86bc7c0617d64d07bc8c433a4cf15156: Generated new TSK 0
I20260806 15:54:13.289920 19223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:13.326828 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38209
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:13.445360 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:13.445663 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:13.445750 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:13.449771 19250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:13.449862 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:13.449956 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:13.455391 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:13.456614 19250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:13.457804 19250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:13.467806 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:13.467774 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:13.467855 19250 server_base.cc:1034] running on GCE node
W20260806 15:54:13.467783 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:13.468392 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:13.469100 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:13.470273 19250 hybrid_clock.cc:648] HybridClock initialized: now 1786031653470253 us; error 33 us; skew 500 ppm
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:13.473352 19250 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:13.474285 19250 webserver.cc:460] Webserver started at http://127.13.223.1:42081/ using document root <none> and password file <none>
I20260806 15:54:13.474853 19250 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:13.474915 19250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:13.475164 19250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:13.477397 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "5cecdba2e55443cd931fa914b3a4ffa5"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "95d5ee593ad838026fe5cc3b7a2e3e87"
server_key_iv: "4c039ca2aff7bd9052e968891afebe44"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.477998 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "5cecdba2e55443cd931fa914b3a4ffa5"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "95d5ee593ad838026fe5cc3b7a2e3e87"
server_key_iv: "4c039ca2aff7bd9052e968891afebe44"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.481863 19250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:13.484297 19271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.485652 19250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:54:13.485795 19250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "5cecdba2e55443cd931fa914b3a4ffa5"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "95d5ee593ad838026fe5cc3b7a2e3e87"
server_key_iv: "4c039ca2aff7bd9052e968891afebe44"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.485905 19250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:13.525166 19250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:13.528719 19250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:13.528934 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:13.529664 19250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:13.530557 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:13.530604 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.530644 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:13.530681 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.539649 19250 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:35373
I20260806 15:54:13.539672 19447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:35373 every 8 connection(s)
I20260806 15:54:13.540627 19250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:13.544951 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 19250
I20260806 15:54:13.545126 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:13.545416 14204 external_mini_cluster.cc:1428] Setting key bfffc47310f2122845cfe611500414ad
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:13.556372 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:13.542432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:35839 (local address 127.13.223.62:38209)
0806 15:54:13.542585 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:13.542588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:13.543504 (+ 916us) server_negotiation.cc:407] Connection header received
0806 15:54:13.544929 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:13.544932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:13.544993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:13.545105 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:13.546635 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.547386 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.548606 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.549017 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.549159 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:13.552042 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:13.552048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:13.552052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:13.552085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:13.553779 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:13.554476 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:13.554479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:13.554481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:13.554563 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:13.555002 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:13.555004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:13.555005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:13.555195 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:13.555274 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:13.556001 (+ 727us) server_negotiation.cc:299] Negotiation successful
0806 15:54:13.556218 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:54:13.557498 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:13.542861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38209 (local address 127.13.223.1:35839)
0806 15:54:13.543346 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:13.543392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:13.544706 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:13.545329 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:13.545336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:13.545813 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:13.546488 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:13.546508 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.547537 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.547539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:13.548460 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:13.548470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.549225 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.549232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:13.549477 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:13.550196 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:13.550225 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:13.551865 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:13.553906 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:13.553916 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:13.553936 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:13.554344 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:13.554702 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:13.554709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:13.554713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:13.554903 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:13.555312 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:13.555321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:13.555683 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 15:54:13.555917 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 15:54:13.556287 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260806 15:54:13.558985 19448 heartbeater.cc:347] Connected to a master server at 127.13.223.62:38209
I20260806 15:54:13.559283 19448 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:54:13.559886 19448 heartbeater.cc:511] Master 127.13.223.62:38209 requested a full tablet report, sending...
I20260806 15:54:13.561661 19154 ts_manager.cc:194] Registered new tserver with Master: 5cecdba2e55443cd931fa914b3a4ffa5 (127.13.223.1:35373)
I20260806 15:54:13.563608 19154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:35839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:13.577898 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38209
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:13.691939 19459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:13.692209 19459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:13.692253 19459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:13.696378 19459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:13.696439 19459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:13.696511 19459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:13.701956 19459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:13.702976 19459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:13.703975 19459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:13.717653 19471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:13.717839 19459 server_base.cc:1034] running on GCE node
W20260806 15:54:13.717666 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:13.718051 19468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:13.718554 19459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:13.719372 19459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:13.720597 19459 hybrid_clock.cc:648] HybridClock initialized: now 1786031653720578 us; error 34 us; skew 500 ppm
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:13.723308 19459 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:13.724243 19459 webserver.cc:460] Webserver started at http://127.13.223.2:39295/ using document root <none> and password file <none>
I20260806 15:54:13.724840 19459 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:13.724905 19459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:13.725072 19459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:13.727082 19459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "87a8ea04fa144941b2d7b1b116609c30"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "05fad55ea2133e56f6c200991f756176"
server_key_iv: "c123c6fb3284f26810b2c84bdde06dfa"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.727614 19459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "87a8ea04fa144941b2d7b1b116609c30"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "05fad55ea2133e56f6c200991f756176"
server_key_iv: "c123c6fb3284f26810b2c84bdde06dfa"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.731508 19459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:54:13.734057 19489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.735383 19459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:13.735522 19459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "87a8ea04fa144941b2d7b1b116609c30"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "05fad55ea2133e56f6c200991f756176"
server_key_iv: "c123c6fb3284f26810b2c84bdde06dfa"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.735636 19459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:13.766319 19459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:13.769510 19459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:13.769726 19459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:13.770349 19459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:13.771204 19459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:13.771275 19459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.771346 19459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:13.771397 19459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.780930 19459 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:46327
I20260806 15:54:13.780958 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:46327 every 8 connection(s)
I20260806 15:54:13.781832 19459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:13.785844 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 19459
I20260806 15:54:13.785987 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:13.786324 14204 external_mini_cluster.cc:1428] Setting key 2fd0ff748839147cdce82ab3355f4b5c
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:13.796554 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:13.783678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:57461 (local address 127.13.223.62:38209)
0806 15:54:13.783828 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:13.783832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:13.784653 (+ 821us) server_negotiation.cc:407] Connection header received
0806 15:54:13.786315 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:13.786318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:13.786367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:13.786476 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:13.788034 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.788756 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.790006 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.790235 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.790379 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:13.792979 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:13.792982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:13.792984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:13.793007 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:13.794443 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:13.794983 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:13.794988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:13.794989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:13.795073 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:13.795416 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:13.795421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:13.795423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:13.795610 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:13.795722 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:13.796222 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 15:54:13.796352 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:13.797446 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:13.783973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38209 (local address 127.13.223.2:57461)
0806 15:54:13.784422 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:13.784468 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:13.786055 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:13.786692 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:13.786720 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:13.787225 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:13.787882 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:13.787902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.788880 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.788884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:13.789867 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:13.789877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:13.790541 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:13.790545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:13.790693 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:13.791321 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:13.791338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:13.792813 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:13.794564 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:13.794574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:13.794589 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:13.794877 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:13.795167 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:13.795172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:13.795177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:13.795321 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:13.795721 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:13.795728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:13.795979 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 15:54:13.796169 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 15:54:13.796497 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
I20260806 15:54:13.798782 19672 heartbeater.cc:347] Connected to a master server at 127.13.223.62:38209
I20260806 15:54:13.799043 19672 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:54:13.799569 19672 heartbeater.cc:511] Master 127.13.223.62:38209 requested a full tablet report, sending...
I20260806 15:54:13.800720 19154 ts_manager.cc:194] Registered new tserver with Master: 87a8ea04fa144941b2d7b1b116609c30 (127.13.223.2:46327)
I20260806 15:54:13.801288 19154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:57461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:13.815901 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38209
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:13.928339 19683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:13.928598 19683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:13.928682 19683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:13.932879 19683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:13.932930 19683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:13.933032 19683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:13.938091 19683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:13.939100 19683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:13.940174 19683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:13.949761 19691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:13.949739 19690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:13.949728 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:13.949893 19683 server_base.cc:1034] running on GCE node
I20260806 15:54:13.950379 19683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:13.951040 19683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:13.952203 19683 hybrid_clock.cc:648] HybridClock initialized: now 1786031653952186 us; error 27 us; skew 500 ppm
Aug 06 15:54:13 dist-test-slave-rwrg krb5kdc[19079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:13.954879 19683 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:13.955724 19683 webserver.cc:460] Webserver started at http://127.13.223.3:33639/ using document root <none> and password file <none>
I20260806 15:54:13.956269 19683 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:13.956382 19683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:13.956660 19683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:13.958750 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "b29493e1a3a64c5ea7f74ed90f19693b"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "b37f0d64154b4a9b5c799627cba6ae31"
server_key_iv: "5564ef286f12c3579a1d302508ee8882"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.959296 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "b29493e1a3a64c5ea7f74ed90f19693b"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "b37f0d64154b4a9b5c799627cba6ae31"
server_key_iv: "5564ef286f12c3579a1d302508ee8882"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.963028 19683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:54:13.965518 19706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:13.966784 19683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:13.966918 19683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "b29493e1a3a64c5ea7f74ed90f19693b"
format_stamp: "Formatted at 2026-08-06 15:54:13 on dist-test-slave-rwrg"
server_key: "b37f0d64154b4a9b5c799627cba6ae31"
server_key_iv: "5564ef286f12c3579a1d302508ee8882"
server_key_version: "encryptionkey@0"
I20260806 15:54:13.967028 19683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:14.004956 19683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:14.007874 19683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:14.008088 19683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:14.008777 19683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:14.009624 19683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:14.009670 19683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:14.009742 19683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:14.009785 19683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:14.019768 19683 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:34527
I20260806 15:54:14.019791 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:34527 every 8 connection(s)
I20260806 15:54:14.020666 19683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:14.023114 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 19683
I20260806 15:54:14.023245 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:14.023579 14204 external_mini_cluster.cc:1428] Setting key 9955274e3f6160b17653bc0de18c841b
Aug 06 15:54:14 dist-test-slave-rwrg krb5kdc[19079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031653, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:14.034381 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:14.022562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:36809 (local address 127.13.223.62:38209)
0806 15:54:14.022723 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:14.022726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:14.023611 (+ 885us) server_negotiation.cc:407] Connection header received
0806 15:54:14.025160 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:14.025163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:14.025223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:14.025331 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:14.026549 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:14.027115 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:14.028260 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:14.028482 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:14.028648 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:14.031158 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:14.031161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:14.031163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:14.031186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:14.032471 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:14.033032 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:14.033035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:14.033036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:14.033127 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:14.033439 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:14.033441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:14.033442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:14.033613 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:14.033695 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:14.034157 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 15:54:14.034257 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 15:54:14.035249 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:14.022891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38209 (local address 127.13.223.3:36809)
0806 15:54:14.023422 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:14.023484 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:14.024946 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:14.025504 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:14.025511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:14.025940 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:14.026409 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:14.026422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:14.027245 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:14.027253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:14.028133 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:14.028142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:14.028690 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:14.028694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:14.028893 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:14.029461 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:14.029477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:14.030983 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:14.032643 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:14.032648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:14.032661 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:14.032937 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:14.033213 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:14.033215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:14.033216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:14.033351 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:14.033710 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:14.033716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:14.033942 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 15:54:14.034101 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 15:54:14.034354 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260806 15:54:14.036586 19870 heartbeater.cc:347] Connected to a master server at 127.13.223.62:38209
I20260806 15:54:14.036846 19870 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:14.037355 19870 heartbeater.cc:511] Master 127.13.223.62:38209 requested a full tablet report, sending...
I20260806 15:54:14.038358 19154 ts_manager.cc:194] Registered new tserver with Master: b29493e1a3a64c5ea7f74ed90f19693b (127.13.223.3:34527)
I20260806 15:54:14.038916 19154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:36809
I20260806 15:54:14.049662 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 15:54:14 dist-test-slave-rwrg krb5kdc[19079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 15:54:14 dist-test-slave-rwrg krb5kdc[19079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:14.079411 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:14.070321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.13.223.62:38209)
0806 15:54:14.070483 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:14.070489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:14.070612 (+ 123us) server_negotiation.cc:407] Connection header received
0806 15:54:14.070871 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:14.070874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:14.070937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:14.071018 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:14.071842 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:14.072373 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:14.073270 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:14.073503 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:14.073637 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:14.076092 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:14.076095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:14.076097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:14.076122 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:14.077673 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:14.078145 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:14.078148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:14.078149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:14.078241 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:14.078593 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:14.078596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:14.078598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:14.078833 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:14.078945 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:14.079173 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 15:54:14.079282 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 15:54:14.079900 19154 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44714
I20260806 15:54:14.080976 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 19250
I20260806 15:54:14.098363 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 19459
I20260806 15:54:14.110185 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 19683
I20260806 15:54:14.133324 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 19096
2026-08-06T15:54:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2453 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:14 dist-test-slave-rwrg krb5kdc[19889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:14 dist-test-slave-rwrg krb5kdc[19889](info): set up 2 sockets
Aug 06 15:54:14 dist-test-slave-rwrg krb5kdc[19889](info): commencing operation
krb5kdc: starting...
W20260806 15:54:15.387473 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:15 dist-test-slave-rwrg krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:15.515698 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:37983
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:37983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:15.629981 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:15.630295 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:15.630365 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:15.634445 19919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:15.634536 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:15.634573 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:15.634611 19919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:15.634642 19919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:15.639873 19919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:37983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:15.641073 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:15.642125 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:15.651160 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:15.651167 19928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:15.651177 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:15.651219 19919 server_base.cc:1034] running on GCE node
I20260806 15:54:15.652067 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:15.653235 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:15.654414 19919 hybrid_clock.cc:648] HybridClock initialized: now 1786031655654384 us; error 42 us; skew 500 ppm
Aug 06 15:54:15 dist-test-slave-rwrg krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031655, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:15.657413 19919 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:15.658427 19919 webserver.cc:460] Webserver started at http://127.13.223.62:36137/ using document root <none> and password file <none>
I20260806 15:54:15.659018 19919 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:15.659119 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:15.659363 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:15.661561 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "f840e2b91db14da6ac7f758e8ddfd84f"
format_stamp: "Formatted at 2026-08-06 15:54:15 on dist-test-slave-rwrg"
server_key: "75e2f691e1671054a16bff4621784385"
server_key_iv: "e04c41bad5b14eb03262f6811eeb4174"
server_key_version: "encryptionkey@0"
I20260806 15:54:15.662177 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "f840e2b91db14da6ac7f758e8ddfd84f"
format_stamp: "Formatted at 2026-08-06 15:54:15 on dist-test-slave-rwrg"
server_key: "75e2f691e1671054a16bff4621784385"
server_key_iv: "e04c41bad5b14eb03262f6811eeb4174"
server_key_version: "encryptionkey@0"
I20260806 15:54:15.666244 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:15.669068 19941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:15.670538 19919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:15.670754 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "f840e2b91db14da6ac7f758e8ddfd84f"
format_stamp: "Formatted at 2026-08-06 15:54:15 on dist-test-slave-rwrg"
server_key: "75e2f691e1671054a16bff4621784385"
server_key_iv: "e04c41bad5b14eb03262f6811eeb4174"
server_key_version: "encryptionkey@0"
I20260806 15:54:15.670889 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:15.714607 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:15.717728 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:15.717947 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:15.725311 19919 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:37983
I20260806 15:54:15.725387 20024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:37983 every 8 connection(s)
I20260806 15:54:15.726347 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:15.729307 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:15.732774 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 19919
I20260806 15:54:15.732941 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:15.733208 14204 external_mini_cluster.cc:1428] Setting key 5fc8dcbbcb4d3a7e8b41d56c0b5269af
I20260806 15:54:15.735387 20026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f: Bootstrap starting.
I20260806 15:54:15.737668 20026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:15.738430 20026 log.cc:824] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:15.740276 20026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f: No bootstrap required, opened a new log
Aug 06 15:54:15 dist-test-slave-rwrg krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:15.744645 20026 raft_consensus.cc:348] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37983 } }
I20260806 15:54:15.744866 20026 raft_consensus.cc:374] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:15.744925 20026 raft_consensus.cc:729] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f840e2b91db14da6ac7f758e8ddfd84f, State: Initialized, Role: FOLLOWER
I20260806 15:54:15.745402 20026 consensus_queue.cc:260] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37983 } }
I20260806 15:54:15.745558 20026 raft_consensus.cc:388] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:15.745628 20026 raft_consensus.cc:482] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:15.745740 20026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:15.746641 20026 raft_consensus.cc:504] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37983 } }
I20260806 15:54:15.746986 20026 leader_election.cc:302] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f840e2b91db14da6ac7f758e8ddfd84f; no voters:
I20260806 15:54:15.747334 20026 leader_election.cc:288] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:15.747411 20032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:15.747663 20032 raft_consensus.cc:686] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [term 1 LEADER]: Becoming Leader. State: Replica: f840e2b91db14da6ac7f758e8ddfd84f, State: Running, Role: LEADER
I20260806 15:54:15.748008 20032 consensus_queue.cc:237] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37983 } }
I20260806 15:54:15.748174 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:15.734720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.13.223.62:37983)
0806 15:54:15.735184 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:15.735199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:15.735237 (+ 38us) server_negotiation.cc:407] Connection header received
0806 15:54:15.735927 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:15.735946 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:15.736280 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:15.736694 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:15.737538 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:15.738668 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:15.739524 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:15.740000 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:15.740187 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:15.743025 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:15.743030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:15.743043 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:15.743077 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:15.744837 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:15.745321 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:15.745327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:15.745331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:15.745446 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:15.745754 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:15.745758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:15.745760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:15.746218 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:15.746441 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:15.746728 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 15:54:15.747066 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260806 15:54:15.748754 20026 sys_catalog.cc:564] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:15.750823 20034 sys_catalog.cc:455] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f840e2b91db14da6ac7f758e8ddfd84f. Latest consensus state: current_term: 1 leader_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37983 } } }
I20260806 15:54:15.750964 20034 sys_catalog.cc:458] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:15.751291 20033 sys_catalog.cc:455] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f840e2b91db14da6ac7f758e8ddfd84f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37983 } } }
I20260806 15:54:15.751374 20054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:15.751431 20033 sys_catalog.cc:458] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:15.754220 20054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:15.759819 20054 catalog_manager.cc:1347] Generated new cluster ID: 25933e864c414f79b3bce13f727b4f63
I20260806 15:54:15.759893 20054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:15.778012 20054 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:15.779331 20054 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:15.786700 20054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f840e2b91db14da6ac7f758e8ddfd84f: Generated new TSK 0
I20260806 15:54:15.787532 20054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:15.823160 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:37983
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:15.938295 20065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:15.938635 20065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:15.938684 20065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:15.942924 20065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:15.942996 20065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:15.943102 20065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:15.948592 20065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:15.949934 20065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:15.951073 20065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:15.961140 20078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:15.961140 20071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:15.961143 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:15.961455 20065 server_base.cc:1034] running on GCE node
I20260806 15:54:15.961894 20065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:15.962738 20065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:15.963938 20065 hybrid_clock.cc:648] HybridClock initialized: now 1786031655963909 us; error 42 us; skew 500 ppm
Aug 06 15:54:15 dist-test-slave-rwrg krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031655, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:15.967211 20065 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:15.968219 20065 webserver.cc:460] Webserver started at http://127.13.223.1:42863/ using document root <none> and password file <none>
I20260806 15:54:15.968864 20065 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:15.968959 20065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:15.969197 20065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:15.971302 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "e6216fdff261490490c4d00aaa843da9"
format_stamp: "Formatted at 2026-08-06 15:54:15 on dist-test-slave-rwrg"
server_key: "2e0e1bfe01387a51bd9a59ed1a0b46f0"
server_key_iv: "486e8cc6dd152445f0eceb5411da1612"
server_key_version: "encryptionkey@0"
I20260806 15:54:15.971865 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "e6216fdff261490490c4d00aaa843da9"
format_stamp: "Formatted at 2026-08-06 15:54:15 on dist-test-slave-rwrg"
server_key: "2e0e1bfe01387a51bd9a59ed1a0b46f0"
server_key_iv: "486e8cc6dd152445f0eceb5411da1612"
server_key_version: "encryptionkey@0"
I20260806 15:54:15.975888 20065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:15.978391 20099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:15.979647 20065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:15.979799 20065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "e6216fdff261490490c4d00aaa843da9"
format_stamp: "Formatted at 2026-08-06 15:54:15 on dist-test-slave-rwrg"
server_key: "2e0e1bfe01387a51bd9a59ed1a0b46f0"
server_key_iv: "486e8cc6dd152445f0eceb5411da1612"
server_key_version: "encryptionkey@0"
I20260806 15:54:15.979916 20065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:15.993623 20065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:15.996690 20065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:15.996889 20065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:15.997478 20065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:15.998310 20065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:15.998382 20065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:15.998459 20065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:15.998507 20065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.007439 20065 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:34473
I20260806 15:54:16.007462 20267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:34473 every 8 connection(s)
I20260806 15:54:16.008322 20065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:16.010504 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 20065
I20260806 15:54:16.010620 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:16.010913 14204 external_mini_cluster.cc:1428] Setting key 042431d42b12507b97b073c730216cda
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031655, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:54:16.022920 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.010214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:55687 (local address 127.13.223.62:37983)
0806 15:54:16.010371 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:16.010377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:16.011355 (+ 978us) server_negotiation.cc:407] Connection header received
0806 15:54:16.012843 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:16.012846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:16.012898 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:16.012979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:16.014147 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.014696 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.015858 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.016100 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.016218 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.019097 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:16.019100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:16.019102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:16.019126 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:16.020767 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.021283 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.021287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.021289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.021380 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.021749 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.021753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.021755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.021950 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:16.022072 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:16.022611 (+ 539us) server_negotiation.cc:299] Negotiation successful
0806 15:54:16.022741 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:54:16.023738 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.010672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:37983 (local address 127.13.223.1:55687)
0806 15:54:16.011184 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:16.011238 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:16.012636 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:16.013159 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:16.013166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:16.013522 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:16.013988 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:16.014003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.014831 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.014835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:16.015722 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:16.015731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.016261 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.016265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:16.016436 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.017112 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:16.017129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:16.018908 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:16.020892 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:16.020898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:16.020912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:16.021174 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:16.021476 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:16.021479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:16.021481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:16.021646 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:16.022062 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:16.022069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:16.022323 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 15:54:16.022525 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 15:54:16.022865 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260806 15:54:16.025031 20268 heartbeater.cc:347] Connected to a master server at 127.13.223.62:37983
I20260806 15:54:16.025316 20268 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:16.025924 20268 heartbeater.cc:511] Master 127.13.223.62:37983 requested a full tablet report, sending...
I20260806 15:54:16.027647 19970 ts_manager.cc:194] Registered new tserver with Master: e6216fdff261490490c4d00aaa843da9 (127.13.223.1:34473)
I20260806 15:54:16.029426 19970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:55687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:16.040624 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:37983
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:16.152714 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:16.152964 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:16.153007 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:16.156913 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:16.156957 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:16.157011 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:16.162132 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:16.163156 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:16.164167 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:16.174624 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:16.174578 20289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:16.174638 20275 server_base.cc:1034] running on GCE node
W20260806 15:54:16.174578 20285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:16.175276 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:16.175968 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:16.177145 20275 hybrid_clock.cc:648] HybridClock initialized: now 1786031656177118 us; error 51 us; skew 500 ppm
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:16.180092 20275 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:16.181227 20275 webserver.cc:460] Webserver started at http://127.13.223.2:32823/ using document root <none> and password file <none>
I20260806 15:54:16.181854 20275 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:16.181932 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:16.182188 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:16.184635 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "409c8c4cdd964abab593b6360ee0a599"
format_stamp: "Formatted at 2026-08-06 15:54:16 on dist-test-slave-rwrg"
server_key: "323566b7a5334b530c9b4dda29bfc8cf"
server_key_iv: "4e28e9a2902055a9fc69f0ed278e9e76"
server_key_version: "encryptionkey@0"
I20260806 15:54:16.185290 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "409c8c4cdd964abab593b6360ee0a599"
format_stamp: "Formatted at 2026-08-06 15:54:16 on dist-test-slave-rwrg"
server_key: "323566b7a5334b530c9b4dda29bfc8cf"
server_key_iv: "4e28e9a2902055a9fc69f0ed278e9e76"
server_key_version: "encryptionkey@0"
I20260806 15:54:16.189441 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:16.192137 20298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.193542 20275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:54:16.193655 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "409c8c4cdd964abab593b6360ee0a599"
format_stamp: "Formatted at 2026-08-06 15:54:16 on dist-test-slave-rwrg"
server_key: "323566b7a5334b530c9b4dda29bfc8cf"
server_key_iv: "4e28e9a2902055a9fc69f0ed278e9e76"
server_key_version: "encryptionkey@0"
I20260806 15:54:16.193790 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:16.216387 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:16.219949 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:16.220240 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:16.220990 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:16.221956 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:16.222039 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.222124 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:16.222175 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.232983 20275 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:44959
I20260806 15:54:16.232998 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:44959 every 8 connection(s)
I20260806 15:54:16.233969 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:16.237879 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 20275
I20260806 15:54:16.237999 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:16.238297 14204 external_mini_cluster.cc:1428] Setting key 181f4c9d8f19617926b167f00395e2e5
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:16.249954 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.235847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:46441 (local address 127.13.223.62:37983)
0806 15:54:16.235980 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:16.235984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:16.236900 (+ 916us) server_negotiation.cc:407] Connection header received
0806 15:54:16.238620 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:16.238623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:16.238708 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:16.238826 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:16.240189 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.240778 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.242441 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.242838 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.242963 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.245776 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:16.245782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:16.245788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:16.245825 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:16.247565 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.248089 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.248094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.248096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.248219 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.248610 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.248616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.248621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.248902 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:16.249017 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:16.249619 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 15:54:16.249740 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:54:16.250890 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.236212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:37983 (local address 127.13.223.2:46441)
0806 15:54:16.236747 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:16.236789 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:16.238377 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:16.239007 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:16.239014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:16.239376 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:16.240013 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:16.240033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.240904 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.240911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:16.242315 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:16.242325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.242986 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.242989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:16.243124 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.243707 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:16.243723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:16.245603 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:16.247687 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:16.247693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:16.247707 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:16.247976 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:16.248315 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:16.248317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:16.248319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:16.248450 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:16.249009 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:16.249014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:16.249345 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 15:54:16.249510 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 15:54:16.249814 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":148,"threads_started":1}
I20260806 15:54:16.252089 20479 heartbeater.cc:347] Connected to a master server at 127.13.223.62:37983
I20260806 15:54:16.252373 20479 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:54:16.252961 20479 heartbeater.cc:511] Master 127.13.223.62:37983 requested a full tablet report, sending...
I20260806 15:54:16.253997 19970 ts_manager.cc:194] Registered new tserver with Master: 409c8c4cdd964abab593b6360ee0a599 (127.13.223.2:44959)
I20260806 15:54:16.254832 19970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:46441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:16.269183 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:37983
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:16.436653 20484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:16.437093 20484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:16.437160 20484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:16.443313 20484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:16.443420 20484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:16.443550 20484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:16.451874 20484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:16.453765 20484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:16.455427 20484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:16.470989 20491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:16.471369 20493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:16.470989 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:16.472019 20484 server_base.cc:1034] running on GCE node
I20260806 15:54:16.473510 20484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:16.475426 20484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:16.479691 20484 hybrid_clock.cc:648] HybridClock initialized: now 1786031656479546 us; error 164 us; skew 500 ppm
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:16.489722 20484 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:16.492383 20484 webserver.cc:460] Webserver started at http://127.13.223.3:36855/ using document root <none> and password file <none>
I20260806 15:54:16.493794 20484 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:16.493968 20484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:16.494499 20484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:16.500270 20484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "8195986420c84af2a4bfd118ba4e1643"
format_stamp: "Formatted at 2026-08-06 15:54:16 on dist-test-slave-rwrg"
server_key: "0e2d67299c166086df72fd7634a5a9c3"
server_key_iv: "8a4867b9ba41ad481db33042845d0daf"
server_key_version: "encryptionkey@0"
I20260806 15:54:16.501965 20484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "8195986420c84af2a4bfd118ba4e1643"
format_stamp: "Formatted at 2026-08-06 15:54:16 on dist-test-slave-rwrg"
server_key: "0e2d67299c166086df72fd7634a5a9c3"
server_key_iv: "8a4867b9ba41ad481db33042845d0daf"
server_key_version: "encryptionkey@0"
I20260806 15:54:16.510125 20484 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260806 15:54:16.514473 20506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.516779 20484 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:16.516973 20484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "8195986420c84af2a4bfd118ba4e1643"
format_stamp: "Formatted at 2026-08-06 15:54:16 on dist-test-slave-rwrg"
server_key: "0e2d67299c166086df72fd7634a5a9c3"
server_key_iv: "8a4867b9ba41ad481db33042845d0daf"
server_key_version: "encryptionkey@0"
I20260806 15:54:16.517159 20484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:16.535706 20484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:16.540937 20484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:16.541239 20484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:16.542232 20484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:16.543565 20484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:16.543673 20484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.543820 20484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:16.543888 20484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:16.564186 20484 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:45603
I20260806 15:54:16.564332 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:45603 every 8 connection(s)
I20260806 15:54:16.565848 20484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:16.568750 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 20484
I20260806 15:54:16.568965 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:16.569331 14204 external_mini_cluster.cc:1428] Setting key 24074d03b63c4aacf558d75c1e8f83e9
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:16.595556 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.570289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:53663 (local address 127.13.223.62:37983)
0806 15:54:16.570528 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:16.570534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:16.572452 (+ 1918us) server_negotiation.cc:407] Connection header received
0806 15:54:16.575344 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:16.575399 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:16.575589 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:16.575817 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:16.578463 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.579268 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.581196 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.581520 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.581711 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.587059 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:16.587065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:16.587069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:16.587120 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:16.591313 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.592662 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.592670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.592672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.592793 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.593521 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.593524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.593525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.593832 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:16.593955 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:16.595202 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0806 15:54:16.595362 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 15:54:16.598382 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.571269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:37983 (local address 127.13.223.3:53663)
0806 15:54:16.572170 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:16.572299 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:16.574858 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:16.576712 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:16.576740 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:16.577449 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:16.578246 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:16.578271 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.579478 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.579482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:16.580961 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:16.580971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.581791 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.581806 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:16.582091 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.583264 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:16.583295 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:16.586707 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:16.591508 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:16.591520 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:16.591564 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:16.592407 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:16.592933 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:16.592937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:16.592952 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:16.593363 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:16.593983 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:16.594003 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:16.594722 (+ 719us) client_negotiation.cc:769] Sending connection context
0806 15:54:16.595154 (+ 432us) client_negotiation.cc:240] Negotiation successful
0806 15:54:16.595931 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":210,"threads_started":1}
I20260806 15:54:16.600608 20683 heartbeater.cc:347] Connected to a master server at 127.13.223.62:37983
I20260806 15:54:16.600992 20683 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:16.601773 20683 heartbeater.cc:511] Master 127.13.223.62:37983 requested a full tablet report, sending...
I20260806 15:54:16.603585 19969 ts_manager.cc:194] Registered new tserver with Master: 8195986420c84af2a4bfd118ba4e1643 (127.13.223.3:45603)
I20260806 15:54:16.605335 19969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:53663
I20260806 15:54:16.615006 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:54:16.672884 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.651666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.13.223.1:34473)
0806 15:54:16.652293 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:16.652316 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:16.652373 (+ 57us) server_negotiation.cc:407] Connection header received
0806 15:54:16.652522 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:16.652534 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:16.653165 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:16.653458 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:16.654499 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.656742 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.658050 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.659285 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.659712 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.665323 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:16.665348 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:16.665377 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:16.665432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:16.668944 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.670164 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.670171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.670173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.670315 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.670955 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.670960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.670961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.671437 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:16.671723 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:16.672265 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 15:54:16.672470 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":215,"threads_started":1}
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:16.697587 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:16.679514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.13.223.62:37983)
0806 15:54:16.679914 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:16.679923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:16.679961 (+ 38us) server_negotiation.cc:407] Connection header received
0806 15:54:16.680161 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:16.680168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:16.680337 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:16.680598 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:16.681640 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.683071 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.684748 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:16.685391 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:16.685631 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:16.690316 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:16.690333 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:16.690339 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:16.690415 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:16.693834 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.694498 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.694506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.694510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.694686 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:16.695057 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:16.695071 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:16.695074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:16.695809 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:16.696079 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:16.696466 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 15:54:16.696813 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
W20260806 15:54:16.698652 19969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42842
I20260806 15:54:16.700243 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 20065
I20260806 15:54:16.752782 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 20275
I20260806 15:54:16.768682 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 20484
I20260806 15:54:16.791219 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 19919
2026-08-06T15:54:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2669 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[20702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[20702](info): set up 2 sockets
Aug 06 15:54:16 dist-test-slave-rwrg krb5kdc[20702](info): commencing operation
krb5kdc: starting...
W20260806 15:54:18.635921 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:18 dist-test-slave-rwrg krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:18.873472 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33613
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:33613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:19.088677 20719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:19.089183 20719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:19.089380 20719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:19.096074 20719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:19.096170 20719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:19.096223 20719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:19.096354 20719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:19.096432 20719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:19.105330 20719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:33613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:19.107285 20719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:19.109269 20719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:19.123274 20725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:19.123313 20728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:19.123317 20724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:19.124028 20719 server_base.cc:1034] running on GCE node
I20260806 15:54:19.125471 20719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:19.127580 20719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:19.130924 20719 hybrid_clock.cc:648] HybridClock initialized: now 1786031659130875 us; error 99 us; skew 500 ppm
Aug 06 15:54:19 dist-test-slave-rwrg krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:19.140962 20719 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:19.142104 20719 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:19.142205 20719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:19.142515 20719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:19.146013 20719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "30545d9375be441a94aee0d83100692f"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "e24509379571909c8e0ac8d7ab5fbb71"
server_key_iv: "983518e2d36f4e733663fd4a0bc1dbc1"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.147075 20719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "30545d9375be441a94aee0d83100692f"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "e24509379571909c8e0ac8d7ab5fbb71"
server_key_iv: "983518e2d36f4e733663fd4a0bc1dbc1"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.153684 20719 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 15:54:19.157886 20735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.160050 20719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260806 15:54:19.160272 20719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "30545d9375be441a94aee0d83100692f"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "e24509379571909c8e0ac8d7ab5fbb71"
server_key_iv: "983518e2d36f4e733663fd4a0bc1dbc1"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.160452 20719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:19.187256 20719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:19.192770 20719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:19.193104 20719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:19.204751 20719 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:33613
I20260806 15:54:19.205241 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:33613 every 8 connection(s)
I20260806 15:54:19.205889 20719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:19.211377 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:19.216329 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 20719
I20260806 15:54:19.216645 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:19.217200 14204 external_mini_cluster.cc:1428] Setting key c86f231dbf5bbab6a420e2fd8175915b
I20260806 15:54:19.228519 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f: Bootstrap starting.
I20260806 15:54:19.239880 20818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:19.241765 20818 log.cc:824] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:19.246112 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f: No bootstrap required, opened a new log
Aug 06 15:54:19 dist-test-slave-rwrg krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:19.254647 20818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30545d9375be441a94aee0d83100692f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33613 } }
I20260806 15:54:19.255318 20818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:19.255447 20818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30545d9375be441a94aee0d83100692f, State: Initialized, Role: FOLLOWER
I20260806 15:54:19.256450 20818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30545d9375be441a94aee0d83100692f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33613 } }
I20260806 15:54:19.256805 20818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:19.256968 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:19.257174 20818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:19.259526 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30545d9375be441a94aee0d83100692f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33613 } }
I20260806 15:54:19.260264 20818 leader_election.cc:302] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30545d9375be441a94aee0d83100692f; no voters:
I20260806 15:54:19.261103 20818 leader_election.cc:288] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:19.261459 20823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:19.261996 20823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [term 1 LEADER]: Becoming Leader. State: Replica: 30545d9375be441a94aee0d83100692f, State: Running, Role: LEADER
I20260806 15:54:19.262173 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:19.219948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.13.223.62:33613)
0806 15:54:19.220786 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:19.220806 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:19.220863 (+ 57us) server_negotiation.cc:407] Connection header received
0806 15:54:19.222180 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:19.222224 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:19.222785 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:19.223479 (+ 694us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:19.226595 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:19.233660 (+ 7065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:19.240792 (+ 7132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:19.241657 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:19.242065 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:19.249076 (+ 7011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:19.249100 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:19.249140 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:19.249226 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:19.254391 (+ 5165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:19.255540 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:19.255562 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:19.255579 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:19.255848 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:19.256574 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:19.256578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:19.256579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:19.257771 (+ 1192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:19.258128 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:19.258770 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 15:54:19.259568 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":219,"threads_started":1}
I20260806 15:54:19.263232 20823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30545d9375be441a94aee0d83100692f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33613 } }
I20260806 15:54:19.264206 20818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:19.265993 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30545d9375be441a94aee0d83100692f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30545d9375be441a94aee0d83100692f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33613 } } }
I20260806 15:54:19.266450 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:19.267001 20824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30545d9375be441a94aee0d83100692f. Latest consensus state: current_term: 1 leader_uuid: "30545d9375be441a94aee0d83100692f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30545d9375be441a94aee0d83100692f" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33613 } } }
I20260806 15:54:19.267203 20824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:19.267900 20839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:19.275637 20839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:19.285899 20839 catalog_manager.cc:1347] Generated new cluster ID: a58653ced344432b922b73f72b99aa41
I20260806 15:54:19.286065 20839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:19.300562 20839 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:19.302021 20839 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:19.320741 20839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f: Generated new TSK 0
I20260806 15:54:19.322446 20839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:19.385586 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33613
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:19.679615 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:19.679955 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:19.680025 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:19.686115 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:19.686205 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:19.686293 20855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:19.692256 20855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:19.693316 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:19.694303 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:19.703985 20863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:19.704093 20868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:19.704061 20866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:19.704252 20855 server_base.cc:1034] running on GCE node
I20260806 15:54:19.704861 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:19.705549 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:19.706773 20855 hybrid_clock.cc:648] HybridClock initialized: now 1786031659706750 us; error 41 us; skew 500 ppm
Aug 06 15:54:19 dist-test-slave-rwrg krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:19.710987 20855 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:19.711709 20855 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:19.711840 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:19.712069 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:19.714318 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "37b808f74a3641b5b75d93247ae0b9f0"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "b38834f608853350730f77f54058106d"
server_key_iv: "47ebaeebf9e5916d06bf5c39129775d7"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.714906 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "37b808f74a3641b5b75d93247ae0b9f0"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "b38834f608853350730f77f54058106d"
server_key_iv: "47ebaeebf9e5916d06bf5c39129775d7"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.718708 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:54:19.721374 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.722613 20855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:19.722731 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "37b808f74a3641b5b75d93247ae0b9f0"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "b38834f608853350730f77f54058106d"
server_key_iv: "47ebaeebf9e5916d06bf5c39129775d7"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.722831 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:19.743724 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:19.747134 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:19.747372 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:19.748047 20855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:19.748999 20855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:19.749048 20855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.749089 20855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:19.749130 20855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.758828 20855 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:34697
I20260806 15:54:19.758850 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:34697 every 8 connection(s)
I20260806 15:54:19.759521 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:19.763626 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 20855
I20260806 15:54:19.763780 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:19.764181 14204 external_mini_cluster.cc:1428] Setting key 99a21edc22af197a59255ddf6a723a47
Aug 06 15:54:19 dist-test-slave-rwrg krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:54:19.775642 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:19.761503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:46971 (local address 127.13.223.62:33613)
0806 15:54:19.761653 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:19.761658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:19.762515 (+ 857us) server_negotiation.cc:407] Connection header received
0806 15:54:19.763864 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:19.763868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:19.763940 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:19.764042 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:19.765601 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:19.766254 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:19.768168 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:19.768598 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:19.768717 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:19.771504 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:19.771509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:19.771513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:19.771555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:19.773408 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:19.773976 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:19.773982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:19.773984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:19.774079 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:19.774440 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:19.774444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:19.774446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:19.774707 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:19.774828 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:19.775302 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 15:54:19.775440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:19.776433 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:19.761899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:33613 (local address 127.13.223.1:46971)
0806 15:54:19.762356 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:19.762400 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:19.763606 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:19.764295 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:19.764306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:19.764850 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:19.765443 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:19.765464 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:19.766406 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:19.766412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:19.767987 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:19.768003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:19.768741 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:19.768744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:19.768881 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:19.769547 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:19.769565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:19.771312 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:19.773579 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:19.773584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:19.773597 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:19.773862 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:19.774192 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:19.774194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:19.774196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:19.774341 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:19.774824 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:19.774830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:19.775062 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 15:54:19.775220 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 15:54:19.775541 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
I20260806 15:54:19.777863 21047 heartbeater.cc:347] Connected to a master server at 127.13.223.62:33613
I20260806 15:54:19.778103 21047 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:54:19.778759 21047 heartbeater.cc:511] Master 127.13.223.62:33613 requested a full tablet report, sending...
I20260806 15:54:19.780288 20762 ts_manager.cc:194] Registered new tserver with Master: 37b808f74a3641b5b75d93247ae0b9f0 (127.13.223.1:34697)
I20260806 15:54:19.781457 20762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:46971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:19.793728 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33613
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:19.908847 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:19.909119 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:19.909197 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:19.913251 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:19.913311 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:19.913414 21057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:19.918907 21057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:19.920007 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:19.921136 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:19.930732 21065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:19.930734 21068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:19.930734 21066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:19.931082 21057 server_base.cc:1034] running on GCE node
I20260806 15:54:19.931638 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:19.932267 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:19.933449 21057 hybrid_clock.cc:648] HybridClock initialized: now 1786031659933401 us; error 61 us; skew 500 ppm
Aug 06 15:54:19 dist-test-slave-rwrg krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:19.936316 21057 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:19.937146 21057 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:19.937211 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:19.937374 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:19.939442 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "6c4650bd8ec64263bd1bea76bc6cad73"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "d7f0fce4b0d334961c61a0caa016989d"
server_key_iv: "7103c709797b4b551a854e1e7eb9f281"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.940033 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "6c4650bd8ec64263bd1bea76bc6cad73"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "d7f0fce4b0d334961c61a0caa016989d"
server_key_iv: "7103c709797b4b551a854e1e7eb9f281"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.943830 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 15:54:19.946352 21078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.947764 21057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:19.947886 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "6c4650bd8ec64263bd1bea76bc6cad73"
format_stamp: "Formatted at 2026-08-06 15:54:19 on dist-test-slave-rwrg"
server_key: "d7f0fce4b0d334961c61a0caa016989d"
server_key_iv: "7103c709797b4b551a854e1e7eb9f281"
server_key_version: "encryptionkey@0"
I20260806 15:54:19.947993 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:19.983999 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:19.987151 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:19.987362 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:19.987993 21057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:19.988943 21057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:19.988992 21057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.989066 21057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:19.989107 21057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:19.998406 21057 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:38751
I20260806 15:54:19.998427 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:38751 every 8 connection(s)
I20260806 15:54:19.999075 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:20.001287 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 21057
I20260806 15:54:20.001448 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:20.001739 14204 external_mini_cluster.cc:1428] Setting key fddad6ce9af91ebc364b8ae08a3cb2b7
Aug 06 15:54:20 dist-test-slave-rwrg krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:20.014564 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.000944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36729 (local address 127.13.223.62:33613)
0806 15:54:20.001105 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.001110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.002166 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 15:54:20.003492 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.003495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.003557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.003648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:20.004873 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.005620 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.007258 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.007502 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.007614 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.010515 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:20.010517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:20.010555 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:20.010584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:20.012410 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:20.013155 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:20.013158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:20.013159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:20.013239 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:20.013555 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:20.013558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:20.013559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:20.013734 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:20.013812 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.014303 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.014441 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:54:20.015571 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.001392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:33613 (local address 127.13.223.2:36729)
0806 15:54:20.001955 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.002009 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.003285 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.003780 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.003786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.004150 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:20.004723 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.004740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.005759 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.005764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.007131 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.007141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.007667 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.007671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.007844 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.008469 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:20.008485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:20.010303 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:20.012591 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:20.012597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:20.012613 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:20.013034 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:20.013324 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:20.013326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:20.013328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:20.013463 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:20.013827 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:20.013831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:20.014059 (+ 228us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.014271 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.014622 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":155,"threads_started":1}
I20260806 15:54:20.016860 21260 heartbeater.cc:347] Connected to a master server at 127.13.223.62:33613
I20260806 15:54:20.017149 21260 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:20.017620 21260 heartbeater.cc:511] Master 127.13.223.62:33613 requested a full tablet report, sending...
I20260806 15:54:20.018710 20762 ts_manager.cc:194] Registered new tserver with Master: 6c4650bd8ec64263bd1bea76bc6cad73 (127.13.223.2:38751)
I20260806 15:54:20.019687 20762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:36729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:20.032349 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33613
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:20.146938 21267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:20.147189 21267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:20.147233 21267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:20.151373 21267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:20.151432 21267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:20.151491 21267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:20.156881 21267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:33613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:20.157887 21267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:20.158859 21267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:20.168359 21277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:20.168380 21276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:20.168452 21280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:20.168903 21267 server_base.cc:1034] running on GCE node
I20260806 15:54:20.169327 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:20.169955 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:20.171142 21267 hybrid_clock.cc:648] HybridClock initialized: now 1786031660171110 us; error 48 us; skew 500 ppm
Aug 06 15:54:20 dist-test-slave-rwrg krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031660, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:20.173996 21267 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:20.174749 21267 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:20.174814 21267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:20.174978 21267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:20.177160 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "e8be9c23f0a044ce889e1b6fa09c0015"
format_stamp: "Formatted at 2026-08-06 15:54:20 on dist-test-slave-rwrg"
server_key: "43ca5b397dd3fa2e50adf5200cba31f1"
server_key_iv: "08ebcbc7babb6e92e5c89bbe30652ffb"
server_key_version: "encryptionkey@0"
I20260806 15:54:20.177680 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "e8be9c23f0a044ce889e1b6fa09c0015"
format_stamp: "Formatted at 2026-08-06 15:54:20 on dist-test-slave-rwrg"
server_key: "43ca5b397dd3fa2e50adf5200cba31f1"
server_key_iv: "08ebcbc7babb6e92e5c89bbe30652ffb"
server_key_version: "encryptionkey@0"
I20260806 15:54:20.181591 21267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:20.184018 21292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:20.185328 21267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:54:20.185447 21267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "e8be9c23f0a044ce889e1b6fa09c0015"
format_stamp: "Formatted at 2026-08-06 15:54:20 on dist-test-slave-rwrg"
server_key: "43ca5b397dd3fa2e50adf5200cba31f1"
server_key_iv: "08ebcbc7babb6e92e5c89bbe30652ffb"
server_key_version: "encryptionkey@0"
I20260806 15:54:20.185554 21267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:20.210948 21267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:20.214280 21267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:20.214495 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:20.215122 21267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:20.216068 21267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:20.216133 21267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:20.216194 21267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:20.216255 21267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:20.225997 21267 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:37909
I20260806 15:54:20.226016 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:37909 every 8 connection(s)
I20260806 15:54:20.226672 21267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:20.229405 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 21267
I20260806 15:54:20.229561 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:20.229892 14204 external_mini_cluster.cc:1428] Setting key 69e0711357f9d0047a87df0a26901bdb
Aug 06 15:54:20 dist-test-slave-rwrg krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031660, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:20.240785 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.228584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:57381 (local address 127.13.223.62:33613)
0806 15:54:20.228738 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.228743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.229812 (+ 1069us) server_negotiation.cc:407] Connection header received
0806 15:54:20.231397 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.231400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.231455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.231532 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:20.232796 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.233357 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.234522 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.234751 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.234866 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.237344 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:20.237349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:20.237357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:20.237381 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:20.238659 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:20.239210 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:20.239213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:20.239214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:20.239288 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:20.239663 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:20.239667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:20.239669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:20.239849 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:20.239958 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.240444 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.240598 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:54:20.241554 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.229023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:33613 (local address 127.13.223.3:57381)
0806 15:54:20.229656 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.229712 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.231181 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.231701 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.231707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.232148 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:20.232651 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.232665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.233503 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.233506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.234392 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.234401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.234900 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.234904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.235081 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.235688 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:20.235704 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:20.237179 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:20.238782 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:20.238787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:20.238802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:20.239097 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:20.239379 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:20.239383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:20.239385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:20.239552 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:20.239957 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:20.239965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:20.240214 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.240388 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.240746 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":182,"threads_started":1}
I20260806 15:54:20.242733 21461 heartbeater.cc:347] Connected to a master server at 127.13.223.62:33613
I20260806 15:54:20.243005 21461 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:20.243580 21461 heartbeater.cc:511] Master 127.13.223.62:33613 requested a full tablet report, sending...
I20260806 15:54:20.244680 20762 ts_manager.cc:194] Registered new tserver with Master: e8be9c23f0a044ce889e1b6fa09c0015 (127.13.223.3:37909)
I20260806 15:54:20.245278 20762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:57381
I20260806 15:54:20.255381 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:54:20.264263 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.257375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.13.223.62:33613)
0806 15:54:20.257529 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.257533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.257689 (+ 156us) server_negotiation.cc:407] Connection header received
0806 15:54:20.257783 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.257785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.257822 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.257878 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:20.258520 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.258998 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.259806 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.260008 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.260085 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.261507 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:20.261510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:20.261512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:20.261534 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:20.262854 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:20.263237 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:20.263240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:20.263241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:20.263311 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:20.263559 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:20.263562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:20.263563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:20.263760 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:20.263870 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.264062 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.264151 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 15:54:20.267175 20762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:20.270255 20762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:20.285334 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.278772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38751 (local address 127.0.0.1:38248)
0806 15:54:20.279107 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.279155 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.279283 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.280428 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.280434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.280459 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:20.280630 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.280636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.282036 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.282041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.283781 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.283788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.284738 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.284742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.284857 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.284896 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.285017 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.285166 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":108,"thread_start_us":110,"threads_started":1}
I20260806 15:54:20.285341 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.278918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.13.223.2:38751)
0806 15:54:20.279460 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.279470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.279492 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:54:20.279598 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.279605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.280205 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.280443 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:20.280759 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.281903 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.283936 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.284608 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.284739 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.284899 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.285076 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.285178 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":179,"threads_started":1}
I20260806 15:54:20.285650 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.279597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37909 (local address 127.0.0.1:58410)
0806 15:54:20.280245 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.280262 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.280359 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.281340 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.281343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.281362 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:20.281460 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.281466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.282833 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.282837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.283805 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.283816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.285051 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.285054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.285283 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.285354 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.285426 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.285515 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":146,"thread_start_us":106,"threads_started":1}
I20260806 15:54:20.286118 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.279861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.13.223.3:37909)
0806 15:54:20.280667 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.280678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.280720 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:54:20.280798 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.280806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.281008 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.281168 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:20.281608 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.282688 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.283967 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.284763 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.284852 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.284983 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.285813 (+ 830us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.285920 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":126,"threads_started":1}
I20260806 15:54:20.286648 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.279759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.13.223.1:34697)
0806 15:54:20.280111 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.280119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.280471 (+ 352us) server_negotiation.cc:407] Connection header received
0806 15:54:20.280576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.280582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.280787 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.280933 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:20.281170 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.282163 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.283905 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.284738 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.284851 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.284984 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.286372 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.286480 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":77,"threads_started":1}
I20260806 15:54:20.286943 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.279589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:34697 (local address 127.0.0.1:60166)
0806 15:54:20.280257 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.280271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.280372 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.280908 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.280911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.280929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:20.281036 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.281042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.282286 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.282289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.283762 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.283775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.285051 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.285054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.285286 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.285354 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.286097 (+ 743us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.286837 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":386,"spinlock_wait_cycles":7936,"thread_start_us":79,"threads_started":1}
I20260806 15:54:20.289638 20958 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c52d7d4134f3e8809aa1b89b1f161 (DEFAULT_TABLE table=test-table [id=5de84990b59a4578bd5fc5fa45f6e886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:20.289642 21157 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c52d7d4134f3e8809aa1b89b1f161 (DEFAULT_TABLE table=test-table [id=5de84990b59a4578bd5fc5fa45f6e886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:20.290338 21362 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c52d7d4134f3e8809aa1b89b1f161 (DEFAULT_TABLE table=test-table [id=5de84990b59a4578bd5fc5fa45f6e886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:20.290903 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c52d7d4134f3e8809aa1b89b1f161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:20.290903 20958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c52d7d4134f3e8809aa1b89b1f161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:20.291249 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c52d7d4134f3e8809aa1b89b1f161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:20.297318 21492 tablet_bootstrap.cc:492] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Bootstrap starting.
I20260806 15:54:20.298784 21495 tablet_bootstrap.cc:492] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Bootstrap starting.
I20260806 15:54:20.299783 21492 tablet_bootstrap.cc:654] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:20.300827 21493 tablet_bootstrap.cc:492] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: Bootstrap starting.
I20260806 15:54:20.301220 21492 log.cc:824] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:20.301329 21495 tablet_bootstrap.cc:654] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:20.302480 21495 log.cc:824] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:20.303097 21493 tablet_bootstrap.cc:654] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:20.303653 21492 tablet_bootstrap.cc:492] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: No bootstrap required, opened a new log
I20260806 15:54:20.303875 21492 ts_tablet_manager.cc:1397] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 15:54:20.303936 21493 log.cc:824] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:20.304347 21495 tablet_bootstrap.cc:492] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: No bootstrap required, opened a new log
I20260806 15:54:20.304556 21495 ts_tablet_manager.cc:1397] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260806 15:54:20.305954 21493 tablet_bootstrap.cc:492] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: No bootstrap required, opened a new log
I20260806 15:54:20.306166 21493 ts_tablet_manager.cc:1397] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 15:54:20.307408 21495 raft_consensus.cc:348] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.307679 21495 raft_consensus.cc:374] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:20.307736 21495 raft_consensus.cc:729] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4650bd8ec64263bd1bea76bc6cad73, State: Initialized, Role: FOLLOWER
I20260806 15:54:20.308218 21495 consensus_queue.cc:260] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.308820 21492 raft_consensus.cc:348] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.309080 21492 raft_consensus.cc:374] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:20.309167 21492 raft_consensus.cc:729] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8be9c23f0a044ce889e1b6fa09c0015, State: Initialized, Role: FOLLOWER
I20260806 15:54:20.309301 21495 ts_tablet_manager.cc:1428] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260806 15:54:20.309602 21492 consensus_queue.cc:260] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.309733 21260 heartbeater.cc:503] Master 127.13.223.62:33613 was elected leader, sending a full tablet report...
I20260806 15:54:20.310222 21461 heartbeater.cc:503] Master 127.13.223.62:33613 was elected leader, sending a full tablet report...
I20260806 15:54:20.310551 21492 ts_tablet_manager.cc:1428] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 15:54:20.310871 21493 raft_consensus.cc:348] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.311154 21493 raft_consensus.cc:374] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:20.311231 21493 raft_consensus.cc:729] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b808f74a3641b5b75d93247ae0b9f0, State: Initialized, Role: FOLLOWER
I20260806 15:54:20.311653 21493 consensus_queue.cc:260] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.312304 21047 heartbeater.cc:503] Master 127.13.223.62:33613 was elected leader, sending a full tablet report...
I20260806 15:54:20.312700 21493 ts_tablet_manager.cc:1428] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 15:54:20.469293 21500 raft_consensus.cc:482] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:20.469511 21500 raft_consensus.cc:504] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.470749 21500 leader_election.cc:288] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c4650bd8ec64263bd1bea76bc6cad73 (127.13.223.2:38751), 37b808f74a3641b5b75d93247ae0b9f0 (127.13.223.1:34697)
I20260806 15:54:20.476250 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.470936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38751 (local address 127.13.223.3:40785)
0806 15:54:20.471104 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.471117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.471192 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.471673 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.471676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.471700 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:20.471833 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.471839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.472961 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.472965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.474608 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.474619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.475654 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.475656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.475824 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.475886 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.475985 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.476087 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":315}
I20260806 15:54:20.476261 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.471428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:57503 (local address 127.13.223.1:34697)
0806 15:54:20.471597 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.471601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.471616 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:20.471661 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.471663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.471703 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.471769 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:20.472060 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.472834 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.474771 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.475498 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.475638 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.475695 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.476049 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.476136 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:54:20.476261 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.471166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:40785 (local address 127.13.223.2:38751)
0806 15:54:20.471331 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.471337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.471350 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:54:20.471406 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.471409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.471457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.471537 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:20.471953 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.472828 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.474771 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.475460 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.475667 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.475717 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.476049 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.476133 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 15:54:20.476259 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.471127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:34697 (local address 127.13.223.3:57503)
0806 15:54:20.471470 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.471486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:20.471569 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:20.471806 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:20.471809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:20.471822 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:20.471930 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.471938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.472961 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.472965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.474613 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:20.474619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.475651 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.475654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:20.475808 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.475886 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 15:54:20.475985 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 15:54:20.476086 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":239,"thread_start_us":111,"threads_started":1}
I20260806 15:54:20.476982 21186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c52d7d4134f3e8809aa1b89b1f161" candidate_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" is_pre_election: true
I20260806 15:54:20.477147 20986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c52d7d4134f3e8809aa1b89b1f161" candidate_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b808f74a3641b5b75d93247ae0b9f0" is_pre_election: true
I20260806 15:54:20.477344 21186 raft_consensus.cc:2454] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8be9c23f0a044ce889e1b6fa09c0015 in term 0.
W20260806 15:54:20.477514 21462 tablet.cc:2369] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:20.477530 20986 raft_consensus.cc:2454] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8be9c23f0a044ce889e1b6fa09c0015 in term 0.
I20260806 15:54:20.477926 21294 leader_election.cc:302] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4650bd8ec64263bd1bea76bc6cad73, e8be9c23f0a044ce889e1b6fa09c0015; no voters:
I20260806 15:54:20.478273 21500 raft_consensus.cc:2781] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:20.478359 21500 raft_consensus.cc:482] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:20.478402 21500 raft_consensus.cc:3037] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:20.479419 21500 raft_consensus.cc:504] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.479768 21500 leader_election.cc:288] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [CANDIDATE]: Term 1 election: Requested vote from peers 6c4650bd8ec64263bd1bea76bc6cad73 (127.13.223.2:38751), 37b808f74a3641b5b75d93247ae0b9f0 (127.13.223.1:34697)
I20260806 15:54:20.480082 21186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c52d7d4134f3e8809aa1b89b1f161" candidate_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4650bd8ec64263bd1bea76bc6cad73"
I20260806 15:54:20.480216 21186 raft_consensus.cc:3037] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:20.480194 20986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c52d7d4134f3e8809aa1b89b1f161" candidate_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b808f74a3641b5b75d93247ae0b9f0"
I20260806 15:54:20.480360 20986 raft_consensus.cc:3037] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:20.481356 21186 raft_consensus.cc:2454] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8be9c23f0a044ce889e1b6fa09c0015 in term 1.
I20260806 15:54:20.481494 20986 raft_consensus.cc:2454] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8be9c23f0a044ce889e1b6fa09c0015 in term 1.
I20260806 15:54:20.481657 21294 leader_election.cc:302] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4650bd8ec64263bd1bea76bc6cad73, e8be9c23f0a044ce889e1b6fa09c0015; no voters:
I20260806 15:54:20.481823 21500 raft_consensus.cc:2781] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:20.482054 21500 raft_consensus.cc:686] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 1 LEADER]: Becoming Leader. State: Replica: e8be9c23f0a044ce889e1b6fa09c0015, State: Running, Role: LEADER
I20260806 15:54:20.482420 21500 consensus_queue.cc:237] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } }
I20260806 15:54:20.485688 20762 catalog_manager.cc:5495] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8be9c23f0a044ce889e1b6fa09c0015 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8be9c23f0a044ce889e1b6fa09c0015" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 } health_report { overall_health: UNKNOWN } } }
W20260806 15:54:20.500254 21262 tablet.cc:2369] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:20.510977 21048 tablet.cc:2369] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:20.520684 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.517108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.13.223.3:37909)
0806 15:54:20.517237 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.517241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.517257 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:20.517355 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.517357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.517414 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.517512 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:20.517750 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.518382 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.519211 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:20.519425 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:20.519520 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:20.519787 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:20.520225 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:20.520316 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.520428 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.520480 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 15:54:20.526520 20986 raft_consensus.cc:1261] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0 [term 1 FOLLOWER]: Refusing update from remote peer e8be9c23f0a044ce889e1b6fa09c0015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:20.526518 21186 raft_consensus.cc:1261] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 1 FOLLOWER]: Refusing update from remote peer e8be9c23f0a044ce889e1b6fa09c0015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:20.527246 21500 consensus_queue.cc:1035] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4650bd8ec64263bd1bea76bc6cad73" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:20.527436 21504 consensus_queue.cc:1035] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b808f74a3641b5b75d93247ae0b9f0" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:20.535920 21509 mvcc.cc:205] Tried to move back new op lower bound from 7315585681509236736 to 7315585681338847232. Current Snapshot: MvccSnapshot[applied={T|T < 7315585681509236736}]
I20260806 15:54:20.543557 20762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:20.543754 20762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:20.546309 20762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30545d9375be441a94aee0d83100692f: Sending DeleteTablet for 3 replicas of tablet 9b5c52d7d4134f3e8809aa1b89b1f161
I20260806 15:54:20.547168 20958 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c52d7d4134f3e8809aa1b89b1f161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:20 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:60166
I20260806 15:54:20.547324 21157 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c52d7d4134f3e8809aa1b89b1f161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:20 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:38248
I20260806 15:54:20.547681 21362 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c52d7d4134f3e8809aa1b89b1f161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:20 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:58410
I20260806 15:54:20.548216 21519 tablet_replica.cc:330] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 37b808f74a3641b5b75d93247ae0b9f0: stopping tablet replica
I20260806 15:54:20.548223 21523 tablet_replica.cc:330] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: stopping tablet replica
I20260806 15:54:20.548359 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 20855
I20260806 15:54:20.549041 21521 tablet_replica.cc:330] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: stopping tablet replica
I20260806 15:54:20.549312 21521 raft_consensus.cc:2229] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:20.549584 21521 raft_consensus.cc:2258] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:20.552188 21523 raft_consensus.cc:2229] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:20.552757 21523 raft_consensus.cc:2258] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:20.555483 21521 ts_tablet_manager.cc:1905] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:20.559180 21523 ts_tablet_manager.cc:1905] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:20.559466 21521 ts_tablet_manager.cc:1918] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:20.559592 21521 log.cc:1196] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/9b5c52d7d4134f3e8809aa1b89b1f161
I20260806 15:54:20.559984 21521 ts_tablet_manager.cc:1939] T 9b5c52d7d4134f3e8809aa1b89b1f161 P 6c4650bd8ec64263bd1bea76bc6cad73: Deleting consensus metadata
I20260806 15:54:20.561594 20737 catalog_manager.cc:4841] TS 6c4650bd8ec64263bd1bea76bc6cad73 (127.13.223.2:38751): tablet 9b5c52d7d4134f3e8809aa1b89b1f161 (table test-table [id=5de84990b59a4578bd5fc5fa45f6e886]) successfully deleted
I20260806 15:54:20.562371 21523 ts_tablet_manager.cc:1918] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:20.562472 21523 log.cc:1196] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/9b5c52d7d4134f3e8809aa1b89b1f161
I20260806 15:54:20.562803 21523 ts_tablet_manager.cc:1939] T 9b5c52d7d4134f3e8809aa1b89b1f161 P e8be9c23f0a044ce889e1b6fa09c0015: Deleting consensus metadata
W20260806 15:54:20.563306 20741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:34697 (error 108)
I20260806 15:54:20.564098 20737 catalog_manager.cc:4841] TS e8be9c23f0a044ce889e1b6fa09c0015 (127.13.223.3:37909): tablet 9b5c52d7d4134f3e8809aa1b89b1f161 (table test-table [id=5de84990b59a4578bd5fc5fa45f6e886]) successfully deleted
I20260806 15:54:20.564566 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.564136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:34697 (local address 127.0.0.1:60182)
0806 15:54:20.564327 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:20.564418 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 15:54:20.564718 20741 catalog_manager.cc:4584] TS 37b808f74a3641b5b75d93247ae0b9f0 (127.13.223.1:34697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b5c52d7d4134f3e8809aa1b89b1f161: Network error: Client connection negotiation failed: client connection to 127.13.223.1:34697: connect: Connection refused (error 111)
I20260806 15:54:20.565037 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 21057
I20260806 15:54:20.581562 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 21267
I20260806 15:54:20.591109 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 20719
2026-08-06T15:54:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3788 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T15:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:20Z Disabled control of system clock
I20260806 15:54:20.630769 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39379
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:39379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 15:54:20.754338 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:20.754629 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:20.754670 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:20.758844 21531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:20.758901 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:20.758916 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:20.758934 21531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:20.758948 21531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:20.763708 21531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:39379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:20.764798 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:20.766165 21531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:20.774847 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:20.774883 21540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:20.774957 21531 server_base.cc:1034] running on GCE node
W20260806 15:54:20.774900 21541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:20.775707 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:20.776886 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:20.778105 21531 hybrid_clock.cc:648] HybridClock initialized: now 1786031660778089 us; error 29 us; skew 500 ppm
I20260806 15:54:20.780141 21531 webserver.cc:460] Webserver started at http://127.13.223.62:41161/ using document root <none> and password file <none>
I20260806 15:54:20.780812 21531 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:20.780906 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:20.781143 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:20.783176 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "efd555452688463f9da239b24b5e9e99"
format_stamp: "Formatted at 2026-08-06 15:54:20 on dist-test-slave-rwrg"
server_key: "8a901898ec3dd2a7cc3873472f25ae5f"
server_key_iv: "e6eddeebd30822ee8399ecf93cb3224a"
server_key_version: "encryptionkey@0"
I20260806 15:54:20.783739 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "efd555452688463f9da239b24b5e9e99"
format_stamp: "Formatted at 2026-08-06 15:54:20 on dist-test-slave-rwrg"
server_key: "8a901898ec3dd2a7cc3873472f25ae5f"
server_key_iv: "e6eddeebd30822ee8399ecf93cb3224a"
server_key_version: "encryptionkey@0"
I20260806 15:54:20.787410 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 15:54:20.790089 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:20.791435 21531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:54:20.791621 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "efd555452688463f9da239b24b5e9e99"
format_stamp: "Formatted at 2026-08-06 15:54:20 on dist-test-slave-rwrg"
server_key: "8a901898ec3dd2a7cc3873472f25ae5f"
server_key_iv: "e6eddeebd30822ee8399ecf93cb3224a"
server_key_version: "encryptionkey@0"
I20260806 15:54:20.791751 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:20.796948 21531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:20.797618 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:20.797838 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:20.805707 21531 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:39379
I20260806 15:54:20.805727 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:39379 every 8 connection(s)
I20260806 15:54:20.806818 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:20.806979 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 21531
I20260806 15:54:20.807137 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:20.807552 14204 external_mini_cluster.cc:1428] Setting key a0ba32b2c617f88de612596d050f8475
I20260806 15:54:20.810504 21630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:20.815032 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:20.809157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.13.223.62:39379)
0806 15:54:20.810490 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:20.810602 (+ 112us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:20.810662 (+ 60us) server_negotiation.cc:407] Connection header received
0806 15:54:20.811444 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:20.811463 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:20.811915 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:20.812320 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:20.813622 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:20.813626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:20.813639 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:20.813649 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:20.813704 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:20.813721 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:20.813919 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:20.814002 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 15:54:20.814278 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":713,"threads_started":1}
I20260806 15:54:20.816660 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99: Bootstrap starting.
I20260806 15:54:20.819103 21630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:20.819836 21630 log.cc:824] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:20.821946 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99: No bootstrap required, opened a new log
I20260806 15:54:20.825580 21630 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd555452688463f9da239b24b5e9e99" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39379 } }
I20260806 15:54:20.825839 21630 raft_consensus.cc:374] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:20.825879 21630 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd555452688463f9da239b24b5e9e99, State: Initialized, Role: FOLLOWER
I20260806 15:54:20.826229 21630 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd555452688463f9da239b24b5e9e99" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39379 } }
I20260806 15:54:20.826335 21630 raft_consensus.cc:388] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:20.826373 21630 raft_consensus.cc:482] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:20.826434 21630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:20.827647 21630 raft_consensus.cc:504] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd555452688463f9da239b24b5e9e99" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39379 } }
I20260806 15:54:20.827980 21630 leader_election.cc:302] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd555452688463f9da239b24b5e9e99; no voters:
I20260806 15:54:20.828248 21630 leader_election.cc:288] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:20.828425 21636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:20.828681 21636 raft_consensus.cc:686] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [term 1 LEADER]: Becoming Leader. State: Replica: efd555452688463f9da239b24b5e9e99, State: Running, Role: LEADER
I20260806 15:54:20.828985 21636 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd555452688463f9da239b24b5e9e99" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39379 } }
I20260806 15:54:20.829540 21630 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:20.830778 21637 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd555452688463f9da239b24b5e9e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd555452688463f9da239b24b5e9e99" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39379 } } }
I20260806 15:54:20.830920 21637 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:20.831240 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd555452688463f9da239b24b5e9e99. Latest consensus state: current_term: 1 leader_uuid: "efd555452688463f9da239b24b5e9e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd555452688463f9da239b24b5e9e99" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39379 } } }
I20260806 15:54:20.831336 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:20.831454 21649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:20.834182 21649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:20.839546 21649 catalog_manager.cc:1347] Generated new cluster ID: a521122289754e209cf17f19ce0d2612
I20260806 15:54:20.839624 21649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:20.854001 21649 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:20.854358 21649 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:20.861756 21649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99: Generated new TSK 0
I20260806 15:54:20.862558 21649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:20.873323 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39379
--builtin_ntp_servers=127.13.223.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 15:54:20.985757 21660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:20.985988 21660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:20.986027 21660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:20.990024 21660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:20.990075 21660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:20.990129 21660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:20.994771 21660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:20.995841 21660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:20.997304 21660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:21.007032 21665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.007051 21666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.007051 21668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:21.007234 21660 server_base.cc:1034] running on GCE node
I20260806 15:54:21.007762 21660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:21.008400 21660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:21.009568 21660 hybrid_clock.cc:648] HybridClock initialized: now 1786031661009552 us; error 30 us; skew 500 ppm
I20260806 15:54:21.011430 21660 webserver.cc:460] Webserver started at http://127.13.223.1:35263/ using document root <none> and password file <none>
I20260806 15:54:21.012002 21660 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:21.012065 21660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:21.012290 21660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:21.014374 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "5f7cda8da5ee42548d51be23d887e1f7"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "743599300f611fb4404cc167de581256"
server_key_iv: "e4b5bb3b217e2432ba42b70ed4589fec"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.014925 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "5f7cda8da5ee42548d51be23d887e1f7"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "743599300f611fb4404cc167de581256"
server_key_iv: "e4b5bb3b217e2432ba42b70ed4589fec"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.018836 21660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 15:54:21.021507 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.022696 21660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:21.022933 21660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "5f7cda8da5ee42548d51be23d887e1f7"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "743599300f611fb4404cc167de581256"
server_key_iv: "e4b5bb3b217e2432ba42b70ed4589fec"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.023053 21660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:21.027667 21660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:21.028287 21660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:21.028474 21660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:21.029098 21660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:21.030175 21660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:21.030248 21660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.030311 21660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:21.030342 21660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.039808 21660 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:39605
I20260806 15:54:21.039829 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:39605 every 8 connection(s)
I20260806 15:54:21.040822 21660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:21.047109 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.042776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:37811 (local address 127.13.223.62:39379)
0806 15:54:21.042968 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.042973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.044184 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 15:54:21.045582 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.045585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.045624 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.045699 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.046750 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.046753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.046755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.046767 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.046785 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.046793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.046865 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.046964 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.047001 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:54:21.047974 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.043335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39379 (local address 127.13.223.1:37811)
0806 15:54:21.044029 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.044069 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.045378 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.045860 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.045866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.045952 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.046468 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.046469 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.046510 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.046576 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.046850 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.046855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.046858 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.046979 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.047170 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260806 15:54:21.049183 21853 heartbeater.cc:347] Connected to a master server at 127.13.223.62:39379
I20260806 15:54:21.049427 21853 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:21.049883 21853 heartbeater.cc:511] Master 127.13.223.62:39379 requested a full tablet report, sending...
I20260806 15:54:21.050335 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 21660
I20260806 15:54:21.050453 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:21.050776 14204 external_mini_cluster.cc:1428] Setting key 5e1fb31a254b359e6a66eb4df472387c
I20260806 15:54:21.051594 21574 ts_manager.cc:194] Registered new tserver with Master: 5f7cda8da5ee42548d51be23d887e1f7 (127.13.223.1:39605)
I20260806 15:54:21.052874 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39379
--builtin_ntp_servers=127.13.223.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 15:54:21.174154 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:21.174543 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:21.174625 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:21.182025 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:21.182114 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:21.182229 21858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:21.189805 21858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:21.191272 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:21.193147 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:21.204625 21865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.204634 21864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.204638 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:21.205127 21858 server_base.cc:1034] running on GCE node
I20260806 15:54:21.205513 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:21.206171 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:21.207332 21858 hybrid_clock.cc:648] HybridClock initialized: now 1786031661207316 us; error 29 us; skew 500 ppm
I20260806 15:54:21.209208 21858 webserver.cc:460] Webserver started at http://127.13.223.2:46259/ using document root <none> and password file <none>
I20260806 15:54:21.209856 21858 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:21.209923 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:21.210091 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:21.212025 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "066c99f899704890a8bb0cdcb5244953"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "40d9f90b474298f01dd21c3fa3a8793e"
server_key_iv: "df6820d1567ad5a6ddb52ef35bf39395"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.212574 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "066c99f899704890a8bb0cdcb5244953"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "40d9f90b474298f01dd21c3fa3a8793e"
server_key_iv: "df6820d1567ad5a6ddb52ef35bf39395"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.216008 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 15:54:21.218427 21875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.219641 21858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:21.219769 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "066c99f899704890a8bb0cdcb5244953"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "40d9f90b474298f01dd21c3fa3a8793e"
server_key_iv: "df6820d1567ad5a6ddb52ef35bf39395"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.219892 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:21.224305 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:21.224939 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:21.225126 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:21.225690 21858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:21.226625 21858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:21.226692 21858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.226756 21858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:21.226809 21858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.236363 21858 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:33453
I20260806 15:54:21.236393 22032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:33453 every 8 connection(s)
I20260806 15:54:21.237269 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:21.239977 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 21858
I20260806 15:54:21.240106 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:21.240410 14204 external_mini_cluster.cc:1428] Setting key 6af3d3216d68b2da37f8361589825314
I20260806 15:54:21.242779 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39379
--builtin_ntp_servers=127.13.223.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 15:54:21.243673 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.239068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:46715 (local address 127.13.223.62:39379)
0806 15:54:21.239211 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.239214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.240248 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 15:54:21.241805 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.241809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.241850 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.241935 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.243198 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.243201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.243204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.243215 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.243238 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.243251 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.243343 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.243492 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.243539 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:54:21.244592 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.239430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39379 (local address 127.13.223.2:46715)
0806 15:54:21.240081 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.240135 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.241590 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.242121 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.242132 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.242233 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.242871 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.242875 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.242959 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.243051 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.243332 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.243342 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.243348 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.243536 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.243793 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
I20260806 15:54:21.245697 22033 heartbeater.cc:347] Connected to a master server at 127.13.223.62:39379
I20260806 15:54:21.245949 22033 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:21.246381 22033 heartbeater.cc:511] Master 127.13.223.62:39379 requested a full tablet report, sending...
I20260806 15:54:21.247462 21574 ts_manager.cc:194] Registered new tserver with Master: 066c99f899704890a8bb0cdcb5244953 (127.13.223.2:33453)
W20260806 15:54:21.356935 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:21.357223 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:21.357302 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:21.361428 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:21.361500 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:21.361589 22037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:21.366291 22037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:21.367379 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:21.368774 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:21.378647 22045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.378638 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.378638 22046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:21.379056 22037 server_base.cc:1034] running on GCE node
I20260806 15:54:21.379519 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:21.380204 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:21.381395 22037 hybrid_clock.cc:648] HybridClock initialized: now 1786031661381358 us; error 47 us; skew 500 ppm
I20260806 15:54:21.383330 22037 webserver.cc:460] Webserver started at http://127.13.223.3:34557/ using document root <none> and password file <none>
I20260806 15:54:21.383922 22037 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:21.384016 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:21.384253 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:21.386372 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "4c5c607814424fc7a1f2bdbc3545d4ff"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "6398fdba05f75393433c75c757c3619b"
server_key_iv: "a4b622a9e780a2e32c6e858d72c3ead0"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.386969 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "4c5c607814424fc7a1f2bdbc3545d4ff"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "6398fdba05f75393433c75c757c3619b"
server_key_iv: "a4b622a9e780a2e32c6e858d72c3ead0"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.390997 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:54:21.393702 22062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.395037 22037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:21.395197 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "4c5c607814424fc7a1f2bdbc3545d4ff"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "6398fdba05f75393433c75c757c3619b"
server_key_iv: "a4b622a9e780a2e32c6e858d72c3ead0"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.395336 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:21.400063 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:21.400774 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:21.400957 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:21.401510 22037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:21.402493 22037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:21.402573 22037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.402642 22037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:21.402696 22037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.413703 22037 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:39141
I20260806 15:54:21.413723 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:39141 every 8 connection(s)
I20260806 15:54:21.414633 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:21.420256 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 22037
I20260806 15:54:21.420403 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:21.420768 14204 external_mini_cluster.cc:1428] Setting key 49b2d7902fdd79b969165fed7de94bb1
I20260806 15:54:21.421308 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.416671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:49167 (local address 127.13.223.62:39379)
0806 15:54:21.416826 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.416831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.418068 (+ 1237us) server_negotiation.cc:407] Connection header received
0806 15:54:21.419340 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.419343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.419382 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.419460 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.420737 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.420740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.420742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.420754 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.420793 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.420805 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.420900 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.421115 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.421159 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:21.422281 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.417156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:39379 (local address 127.13.223.3:49167)
0806 15:54:21.417866 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.417925 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.419147 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.419657 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.419663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.419752 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.420380 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.420382 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.420452 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.420573 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.420949 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.420957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.420962 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.421141 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.421457 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260806 15:54:21.423327 22257 heartbeater.cc:347] Connected to a master server at 127.13.223.62:39379
I20260806 15:54:21.423583 22257 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:21.423985 22257 heartbeater.cc:511] Master 127.13.223.62:39379 requested a full tablet report, sending...
I20260806 15:54:21.425364 21574 ts_manager.cc:194] Registered new tserver with Master: 4c5c607814424fc7a1f2bdbc3545d4ff (127.13.223.3:39141)
I20260806 15:54:21.434934 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:54:21.438709 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.436979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.13.223.62:39379)
0806 15:54:21.437136 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.437140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.437299 (+ 159us) server_negotiation.cc:407] Connection header received
0806 15:54:21.437476 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.437486 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.437524 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.437620 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.438312 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.438315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.438318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.438332 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.438355 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.438363 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.438464 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.438534 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.438575 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:54:21.441486 21574 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:21.444621 21574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:21.456633 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.454110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:39141 (local address 127.0.0.1:33356)
0806 15:54:21.454643 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.454657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.454741 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.455423 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.455426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.455440 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.455888 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.455890 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.455900 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.455926 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.456211 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.456220 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.456226 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.456334 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.456426 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260806 15:54:21.456723 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.453440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.13.223.1:39605)
0806 15:54:21.454159 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.454174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.454201 (+ 27us) server_negotiation.cc:407] Connection header received
0806 15:54:21.454259 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.454265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.454466 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.455070 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.456125 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.456132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.456145 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.456157 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.456198 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.456230 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.456344 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.456423 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.456546 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":151,"threads_started":1}
I20260806 15:54:21.457134 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.453292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:39605 (local address 127.0.0.1:42494)
0806 15:54:21.453699 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.453752 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.453861 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.454648 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.454654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.454739 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.455363 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.455367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.455393 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.455461 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.456306 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.456312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.456317 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.456370 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.456420 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":127,"threads_started":1}
I20260806 15:54:21.457348 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.454578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.13.223.3:39141)
0806 15:54:21.454996 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.455007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.455030 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:54:21.455110 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.455115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.455279 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.455410 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.456043 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.456048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.456057 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.456065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.456097 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.456114 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.456236 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.456979 (+ 743us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.457054 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260806 15:54:21.457816 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.453556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:33453 (local address 127.0.0.1:45154)
0806 15:54:21.454285 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.454298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.454367 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.456032 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.456046 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.456060 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.456501 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.456503 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.456537 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.456563 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.457589 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.457592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.457596 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.457669 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.457705 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":103,"threads_started":1}
I20260806 15:54:21.459995 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.454589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.13.223.2:33453)
0806 15:54:21.455038 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.455048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.455070 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:54:21.455125 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.455129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.455278 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.455410 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.457067 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.457076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.457089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.457104 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.457148 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.457330 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.457628 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.457679 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.457742 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":135,"threads_started":1}
I20260806 15:54:21.459702 22144 tablet_service.cc:1467] Processing CreateTablet for tablet ae9d3dd0aebb4b0c8b4d8866827291cb (DEFAULT_TABLE table=test-table [id=14a478a7693e4ceaa1cb00ca4379401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:21.459703 21746 tablet_service.cc:1467] Processing CreateTablet for tablet ae9d3dd0aebb4b0c8b4d8866827291cb (DEFAULT_TABLE table=test-table [id=14a478a7693e4ceaa1cb00ca4379401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:21.460824 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9d3dd0aebb4b0c8b4d8866827291cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:21.460824 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9d3dd0aebb4b0c8b4d8866827291cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:21.462638 21941 tablet_service.cc:1467] Processing CreateTablet for tablet ae9d3dd0aebb4b0c8b4d8866827291cb (DEFAULT_TABLE table=test-table [id=14a478a7693e4ceaa1cb00ca4379401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:21.463647 21941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9d3dd0aebb4b0c8b4d8866827291cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:21.467063 22283 tablet_bootstrap.cc:492] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Bootstrap starting.
I20260806 15:54:21.468590 22286 tablet_bootstrap.cc:492] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: Bootstrap starting.
I20260806 15:54:21.469516 22283 tablet_bootstrap.cc:654] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:21.470266 22283 log.cc:824] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:21.470626 22287 tablet_bootstrap.cc:492] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Bootstrap starting.
I20260806 15:54:21.470746 22286 tablet_bootstrap.cc:654] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:21.471501 22286 log.cc:824] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:21.472350 22283 tablet_bootstrap.cc:492] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: No bootstrap required, opened a new log
I20260806 15:54:21.472570 22283 ts_tablet_manager.cc:1397] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:54:21.472826 22287 tablet_bootstrap.cc:654] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:21.473369 22286 tablet_bootstrap.cc:492] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: No bootstrap required, opened a new log
I20260806 15:54:21.473572 22286 ts_tablet_manager.cc:1397] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 15:54:21.473630 22287 log.cc:824] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:21.475561 22287 tablet_bootstrap.cc:492] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: No bootstrap required, opened a new log
I20260806 15:54:21.475749 22287 ts_tablet_manager.cc:1397] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 15:54:21.476764 22283 raft_consensus.cc:348] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.477010 22283 raft_consensus.cc:374] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:21.477074 22283 raft_consensus.cc:729] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5c607814424fc7a1f2bdbc3545d4ff, State: Initialized, Role: FOLLOWER
I20260806 15:54:21.477525 22283 consensus_queue.cc:260] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.478197 22257 heartbeater.cc:503] Master 127.13.223.62:39379 was elected leader, sending a full tablet report...
I20260806 15:54:21.478058 22286 raft_consensus.cc:348] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.478313 22286 raft_consensus.cc:374] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:21.478370 22286 raft_consensus.cc:729] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7cda8da5ee42548d51be23d887e1f7, State: Initialized, Role: FOLLOWER
I20260806 15:54:21.478557 22283 ts_tablet_manager.cc:1428] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:54:21.478790 22286 consensus_queue.cc:260] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.479496 21853 heartbeater.cc:503] Master 127.13.223.62:39379 was elected leader, sending a full tablet report...
I20260806 15:54:21.480086 22286 ts_tablet_manager.cc:1428] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 15:54:21.480974 22287 raft_consensus.cc:348] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.481258 22287 raft_consensus.cc:374] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:21.481307 22287 raft_consensus.cc:729] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066c99f899704890a8bb0cdcb5244953, State: Initialized, Role: FOLLOWER
I20260806 15:54:21.481689 22287 consensus_queue.cc:260] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.482349 22033 heartbeater.cc:503] Master 127.13.223.62:39379 was elected leader, sending a full tablet report...
I20260806 15:54:21.482647 22287 ts_tablet_manager.cc:1428] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 15:54:21.486900 22295 raft_consensus.cc:482] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:21.487056 22295 raft_consensus.cc:504] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.488054 22295 leader_election.cc:288] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f7cda8da5ee42548d51be23d887e1f7 (127.13.223.1:39605), 4c5c607814424fc7a1f2bdbc3545d4ff (127.13.223.3:39141)
W20260806 15:54:21.488202 22035 tablet.cc:2369] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:21.490074 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.488236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:39605 (local address 127.13.223.2:35437)
0806 15:54:21.488451 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.488468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.488587 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.488864 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.488867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.488882 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.489451 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.489452 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.489462 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.489506 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.489855 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.489858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.489861 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.489938 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.489972 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 15:54:21.490329 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.488688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36489 (local address 127.13.223.3:39141)
0806 15:54:21.489001 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.489008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.489024 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:21.489176 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.489179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.489209 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.489281 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.489949 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.489952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.489955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.489966 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.489983 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.489999 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.490105 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.490170 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.490207 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260806 15:54:21.490350 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.488482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:39141 (local address 127.13.223.2:36489)
0806 15:54:21.488941 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.488959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.489055 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.489328 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.489333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.489349 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:21.489800 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:21.489803 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:21.489813 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:21.489836 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:21.490092 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:21.490095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:21.490098 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 15:54:21.490175 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 15:54:21.490207 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":114,"threads_started":1}
I20260806 15:54:21.490720 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.488353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:35437 (local address 127.13.223.1:39605)
0806 15:54:21.488475 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.488480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.488539 (+ 59us) server_negotiation.cc:407] Connection header received
0806 15:54:21.488709 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.488712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.488749 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.488847 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.489647 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.489654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.489660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.489677 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.489701 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.489714 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.489807 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.490560 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.490599 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:54:21.491448 21779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d3dd0aebb4b0c8b4d8866827291cb" candidate_uuid: "066c99f899704890a8bb0cdcb5244953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7cda8da5ee42548d51be23d887e1f7" is_pre_election: true
I20260806 15:54:21.491447 22182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d3dd0aebb4b0c8b4d8866827291cb" candidate_uuid: "066c99f899704890a8bb0cdcb5244953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" is_pre_election: true
I20260806 15:54:21.491729 21779 raft_consensus.cc:2454] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066c99f899704890a8bb0cdcb5244953 in term 0.
I20260806 15:54:21.491729 22182 raft_consensus.cc:2454] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066c99f899704890a8bb0cdcb5244953 in term 0.
I20260806 15:54:21.492189 21880 leader_election.cc:302] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066c99f899704890a8bb0cdcb5244953, 5f7cda8da5ee42548d51be23d887e1f7; no voters:
I20260806 15:54:21.492417 22295 raft_consensus.cc:2781] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:21.492527 22295 raft_consensus.cc:482] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:21.492578 22295 raft_consensus.cc:3037] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:21.493702 22295 raft_consensus.cc:504] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.494025 22295 leader_election.cc:288] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [CANDIDATE]: Term 1 election: Requested vote from peers 5f7cda8da5ee42548d51be23d887e1f7 (127.13.223.1:39605), 4c5c607814424fc7a1f2bdbc3545d4ff (127.13.223.3:39141)
I20260806 15:54:21.494365 21779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d3dd0aebb4b0c8b4d8866827291cb" candidate_uuid: "066c99f899704890a8bb0cdcb5244953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7cda8da5ee42548d51be23d887e1f7"
I20260806 15:54:21.494429 22182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d3dd0aebb4b0c8b4d8866827291cb" candidate_uuid: "066c99f899704890a8bb0cdcb5244953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff"
I20260806 15:54:21.494489 21779 raft_consensus.cc:3037] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:21.494530 22182 raft_consensus.cc:3037] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:21.495795 22182 raft_consensus.cc:2454] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066c99f899704890a8bb0cdcb5244953 in term 1.
I20260806 15:54:21.495795 21779 raft_consensus.cc:2454] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066c99f899704890a8bb0cdcb5244953 in term 1.
I20260806 15:54:21.496070 21880 leader_election.cc:302] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066c99f899704890a8bb0cdcb5244953, 4c5c607814424fc7a1f2bdbc3545d4ff; no voters:
I20260806 15:54:21.496240 22295 raft_consensus.cc:2781] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:21.496474 22295 raft_consensus.cc:686] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 1 LEADER]: Becoming Leader. State: Replica: 066c99f899704890a8bb0cdcb5244953, State: Running, Role: LEADER
I20260806 15:54:21.496838 22295 consensus_queue.cc:237] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } }
I20260806 15:54:21.499970 21575 catalog_manager.cc:5495] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 reported cstate change: term changed from 0 to 1, leader changed from <none> to 066c99f899704890a8bb0cdcb5244953 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "066c99f899704890a8bb0cdcb5244953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "066c99f899704890a8bb0cdcb5244953" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 33453 } health_report { overall_health: HEALTHY } } }
I20260806 15:54:21.521257 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.519667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.13.223.2:33453)
0806 15:54:21.519788 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.519792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.519807 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:21.519926 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.519928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.519984 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.520099 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.520834 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.520837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.520839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.520850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.520872 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.520886 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.520977 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.521072 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.521112 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 15:54:21.527199 22182 raft_consensus.cc:1261] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 1 FOLLOWER]: Refusing update from remote peer 066c99f899704890a8bb0cdcb5244953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:21.527197 21779 raft_consensus.cc:1261] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 1 FOLLOWER]: Refusing update from remote peer 066c99f899704890a8bb0cdcb5244953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:21.527980 22297 consensus_queue.cc:1035] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7cda8da5ee42548d51be23d887e1f7" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:21.528230 22295 consensus_queue.cc:1035] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5c607814424fc7a1f2bdbc3545d4ff" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:21.535831 22303 mvcc.cc:205] Tried to move back new op lower bound from 7315585685608095744 to 7315585685493571584. Current Snapshot: MvccSnapshot[applied={T|T < 7315585685608095744}]
I20260806 15:54:21.538615 22302 mvcc.cc:205] Tried to move back new op lower bound from 7315585685608095744 to 7315585685493571584. Current Snapshot: MvccSnapshot[applied={T|T < 7315585685608095744}]
I20260806 15:54:21.545327 21574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:21.545547 21574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:21.548298 21574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efd555452688463f9da239b24b5e9e99: Sending DeleteTablet for 3 replicas of tablet ae9d3dd0aebb4b0c8b4d8866827291cb
I20260806 15:54:21.549399 21941 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9d3dd0aebb4b0c8b4d8866827291cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:21 UTC) from {username='slave'} at 127.0.0.1:45154
I20260806 15:54:21.549619 21746 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9d3dd0aebb4b0c8b4d8866827291cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:21 UTC) from {username='slave'} at 127.0.0.1:42494
I20260806 15:54:21.550091 22314 tablet_replica.cc:330] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7: stopping tablet replica
I20260806 15:54:21.550038 22144 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9d3dd0aebb4b0c8b4d8866827291cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:21 UTC) from {username='slave'} at 127.0.0.1:33356
I20260806 15:54:21.550609 22313 tablet_replica.cc:330] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: stopping tablet replica
I20260806 15:54:21.550750 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 21660
I20260806 15:54:21.550840 22314 raft_consensus.cc:2229] T ae9d3dd0aebb4b0c8b4d8866827291cb P 5f7cda8da5ee42548d51be23d887e1f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:21.553774 22317 tablet_replica.cc:330] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: stopping tablet replica
I20260806 15:54:21.554136 22317 raft_consensus.cc:2229] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:21.554534 22317 raft_consensus.cc:2258] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:21.554992 22313 raft_consensus.cc:2229] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:21.555500 22313 raft_consensus.cc:2258] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:21.558933 22313 ts_tablet_manager.cc:1905] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:21.559758 22317 ts_tablet_manager.cc:1905] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:21.562440 22317 ts_tablet_manager.cc:1918] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:21.562639 22317 log.cc:1196] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/ae9d3dd0aebb4b0c8b4d8866827291cb
I20260806 15:54:21.563033 22317 ts_tablet_manager.cc:1939] T ae9d3dd0aebb4b0c8b4d8866827291cb P 4c5c607814424fc7a1f2bdbc3545d4ff: Deleting consensus metadata
I20260806 15:54:21.563097 22313 ts_tablet_manager.cc:1918] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:21.563238 22313 log.cc:1196] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/ae9d3dd0aebb4b0c8b4d8866827291cb
I20260806 15:54:21.563720 22313 ts_tablet_manager.cc:1939] T ae9d3dd0aebb4b0c8b4d8866827291cb P 066c99f899704890a8bb0cdcb5244953: Deleting consensus metadata
I20260806 15:54:21.564648 21556 catalog_manager.cc:4841] TS 4c5c607814424fc7a1f2bdbc3545d4ff (127.13.223.3:39141): tablet ae9d3dd0aebb4b0c8b4d8866827291cb (table test-table [id=14a478a7693e4ceaa1cb00ca4379401d]) successfully deleted
I20260806 15:54:21.565198 21557 catalog_manager.cc:4841] TS 066c99f899704890a8bb0cdcb5244953 (127.13.223.2:33453): tablet ae9d3dd0aebb4b0c8b4d8866827291cb (table test-table [id=14a478a7693e4ceaa1cb00ca4379401d]) successfully deleted
W20260806 15:54:21.575479 21556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:39605 (error 108)
I20260806 15:54:21.575527 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 21858
I20260806 15:54:21.577889 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.576404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:39605 (local address 127.0.0.1:42496)
0806 15:54:21.577682 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.577775 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:39605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1148}
W20260806 15:54:21.582089 21556 catalog_manager.cc:4584] TS 5f7cda8da5ee42548d51be23d887e1f7 (127.13.223.1:39605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae9d3dd0aebb4b0c8b4d8866827291cb: Network error: Client connection negotiation failed: client connection to 127.13.223.1:39605: connect: Connection refused (error 111)
I20260806 15:54:21.587545 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 22037
I20260806 15:54:21.600450 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 21531
2026-08-06T15:54:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1004 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 15:54:21.625051 14204 mini_oidc.cc:152] Starting JWKS server
I20260806 15:54:21.626915 14204 webserver.cc:460] Webserver started at https://127.0.0.1:33173/ using document root <none> and password file <none>
I20260806 15:54:21.627116 14204 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 15:54:21.627434 14204 webserver.cc:460] Webserver started at http://127.0.0.1:42723/ using document root <none> and password file <none>
2026-08-06T15:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:21Z Disabled control of system clock
I20260806 15:54:21.638131 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34629
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:45283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34629
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33173/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/testchainca.cert with env {}
W20260806 15:54:21.753449 22326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:21.753729 22326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:21.753800 22326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:21.757822 22326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:21.757879 22326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:21.757894 22326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:21.757911 22326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:21.757964 22326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 15:54:21.757985 22326 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 15:54:21.758015 22326 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33173/jwks.json
I20260806 15:54:21.763302 22326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33173/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:21.764396 22326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:21.765472 22326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:21.774132 22339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.774152 22337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:21.774199 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:21.774214 22326 server_base.cc:1034] running on GCE node
I20260806 15:54:21.775050 22326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:21.776157 22326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:21.777359 22326 hybrid_clock.cc:648] HybridClock initialized: now 1786031661777326 us; error 45 us; skew 500 ppm
I20260806 15:54:21.779332 22326 webserver.cc:460] Webserver started at http://127.13.223.62:38137/ using document root <none> and password file <none>
I20260806 15:54:21.779942 22326 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:21.780038 22326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:21.780265 22326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:21.782330 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "cd991ef594684ee1850eda9191db5337"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "7047e36cf0dbd95e96f4ed3254d154de"
server_key_iv: "bab189220c8d2635a2e864a22d323bbb"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.782893 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "cd991ef594684ee1850eda9191db5337"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "7047e36cf0dbd95e96f4ed3254d154de"
server_key_iv: "bab189220c8d2635a2e864a22d323bbb"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.786654 22326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:54:21.789434 22349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:21.790743 22326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:21.790894 22326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "cd991ef594684ee1850eda9191db5337"
format_stamp: "Formatted at 2026-08-06 15:54:21 on dist-test-slave-rwrg"
server_key: "7047e36cf0dbd95e96f4ed3254d154de"
server_key_iv: "bab189220c8d2635a2e864a22d323bbb"
server_key_version: "encryptionkey@0"
I20260806 15:54:21.791013 22326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:21.811357 22326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:21.812155 22326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:21.812366 22326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:21.819622 22326 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34629
I20260806 15:54:21.819674 22431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34629 every 8 connection(s)
I20260806 15:54:21.820698 22326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:21.823817 22432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:21.824436 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 22326
I20260806 15:54:21.824616 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:21.824913 14204 external_mini_cluster.cc:1428] Setting key 5a6dc946daf1f374bcdec7187efb7ef4
I20260806 15:54:21.829767 22432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337: Bootstrap starting.
I20260806 15:54:21.832006 22432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:21.832908 22432 log.cc:824] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:21.834626 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.826327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.13.223.62:34629)
0806 15:54:21.827041 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.827057 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.827143 (+ 86us) server_negotiation.cc:407] Connection header received
0806 15:54:21.827854 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.827872 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.828316 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.828679 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:21.829110 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:21.830110 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:21.831004 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:21.831482 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:21.831663 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:21.832641 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:21.832646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:21.832659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:21.833072 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:21.833150 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:21.833172 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:21.833255 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:21.833445 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 15:54:21.833702 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260806 15:54:21.835213 22432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337: No bootstrap required, opened a new log
I20260806 15:54:21.837436 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:54:21.839238 22432 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd991ef594684ee1850eda9191db5337" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34629 } }
I20260806 15:54:21.839469 22432 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:21.839552 22432 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd991ef594684ee1850eda9191db5337, State: Initialized, Role: FOLLOWER
I20260806 15:54:21.839988 22432 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd991ef594684ee1850eda9191db5337" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34629 } }
I20260806 15:54:21.840131 22432 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:21.840219 22432 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:21.840314 22432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:21.841249 22432 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd991ef594684ee1850eda9191db5337" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34629 } }
I20260806 15:54:21.841605 22432 leader_election.cc:302] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd991ef594684ee1850eda9191db5337; no voters:
I20260806 15:54:21.841918 22432 leader_election.cc:288] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:21.842044 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:21.842306 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [term 1 LEADER]: Becoming Leader. State: Replica: cd991ef594684ee1850eda9191db5337, State: Running, Role: LEADER
I20260806 15:54:21.842615 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd991ef594684ee1850eda9191db5337" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34629 } }
I20260806 15:54:21.843185 22432 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:21.844310 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd991ef594684ee1850eda9191db5337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd991ef594684ee1850eda9191db5337" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34629 } } }
I20260806 15:54:21.844439 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:21.844851 22450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:21.845299 22442 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd991ef594684ee1850eda9191db5337. Latest consensus state: current_term: 1 leader_uuid: "cd991ef594684ee1850eda9191db5337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd991ef594684ee1850eda9191db5337" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34629 } } }
I20260806 15:54:21.845412 22442 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:21.847993 22450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:21.853312 22450 catalog_manager.cc:1347] Generated new cluster ID: 128ae4a392c44407af8ff3422069c3fa
I20260806 15:54:21.853371 22450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:21.868124 22450 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:21.869357 22450 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:21.877571 22450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd991ef594684ee1850eda9191db5337: Generated new TSK 0
I20260806 15:54:21.878491 22450 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 15:54:21.906721 22480 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 15:54:21.906914 22480 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 15:54:21.907433 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:21.890059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.13.223.62:34629)
0806 15:54:21.890223 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:21.890229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:21.890455 (+ 226us) server_negotiation.cc:407] Connection header received
0806 15:54:21.890695 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:21.890697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:21.890747 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:21.890833 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0806 15:54:21.891153 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:21.891703 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:21.892952 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:21.893179 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:21.893259 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:21.893649 (+ 390us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 15:54:21.907059 (+ 13410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 15:54:21.907309 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 15:54:21.907516 22437 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 15:54:21.909124 22478 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:21.889956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34629 (local address 127.0.0.1:52500)
0806 15:54:21.890356 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:21.890373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:21.890515 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:21.890882 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:21.890885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:21.890894 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 15:54:21.891037 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:21.891044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:21.891824 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:21.891828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:21.892836 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:21.892849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:21.893300 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:21.893304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:21.893402 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:21.893456 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 15:54:21.907308 (+ 13852us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 15:54:21.907563 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.62:34629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.223.62:34629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 15:54:21.910209 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 22326
2026-08-06T15:54:21Z chronyd exiting
I20260806 15:54:21.967844 14204 test_util.cc:175] -----------------------------------------------
I20260806 15:54:21.968050 14204 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786031632208048-14204-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 15:54:21.969355 14204 mini_oidc.cc:152] Starting JWKS server
I20260806 15:54:21.970228 14204 webserver.cc:460] Webserver started at https://127.0.0.1:42843/ using document root <none> and password file <none>
I20260806 15:54:21.970355 14204 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 15:54:21.970619 14204 webserver.cc:460] Webserver started at http://127.0.0.1:42149/ using document root <none> and password file <none>
2026-08-06T15:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:21Z Disabled control of system clock
I20260806 15:54:21.981073 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34211
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/testchainca.cert with env {}
W20260806 15:54:22.091351 22487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:22.091619 22487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:22.091660 22487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:22.095633 22487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:22.095690 22487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:22.095705 22487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:22.095721 22487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:22.095736 22487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 15:54:22.095752 22487 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 15:54:22.095818 22487 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42843/jwks.json
I20260806 15:54:22.101190 22487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:22.102233 22487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:22.103247 22487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:22.111948 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:22.111999 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:22.111941 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:22.112773 22487 server_base.cc:1034] running on GCE node
I20260806 15:54:22.113374 22487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:22.114503 22487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:22.115707 22487 hybrid_clock.cc:648] HybridClock initialized: now 1786031662115690 us; error 38 us; skew 500 ppm
I20260806 15:54:22.117568 22487 webserver.cc:460] Webserver started at http://127.13.223.62:41791/ using document root <none> and password file <none>
I20260806 15:54:22.118184 22487 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:22.118278 22487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:22.118508 22487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:22.120747 22487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "cd2cdfe0c71e405db8375ead3af2d9c1"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "05303aadbcd81cc60a734835617fa491"
server_key_iv: "e5d9455bd7775631c322a9bd35b33575"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.121348 22487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "cd2cdfe0c71e405db8375ead3af2d9c1"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "05303aadbcd81cc60a734835617fa491"
server_key_iv: "e5d9455bd7775631c322a9bd35b33575"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.125303 22487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:54:22.127748 22508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:22.129076 22487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:22.129241 22487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "cd2cdfe0c71e405db8375ead3af2d9c1"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "05303aadbcd81cc60a734835617fa491"
server_key_iv: "e5d9455bd7775631c322a9bd35b33575"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.129377 22487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:22.143944 22487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:22.144631 22487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:22.144830 22487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:22.151723 22487 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34211
I20260806 15:54:22.151728 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34211 every 8 connection(s)
I20260806 15:54:22.152686 22487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:22.155916 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:22.157052 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 22487
I20260806 15:54:22.157202 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:22.157537 14204 external_mini_cluster.cc:1428] Setting key 2f1a108796f236ec2059621f4b558ebb
I20260806 15:54:22.162245 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1: Bootstrap starting.
I20260806 15:54:22.164426 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:22.165324 22591 log.cc:824] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:22.167327 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:22.158944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.13.223.62:34211)
0806 15:54:22.159652 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:22.159670 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:22.159709 (+ 39us) server_negotiation.cc:407] Connection header received
0806 15:54:22.160367 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:22.160383 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:22.160912 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:22.161257 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:22.161706 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.162782 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.163732 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.164236 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.164440 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.165402 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:22.165408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:22.165421 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:22.165938 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:22.166017 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:22.166047 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:22.166151 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:22.166326 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 15:54:22.166620 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260806 15:54:22.167445 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1: No bootstrap required, opened a new log
I20260806 15:54:22.170495 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:54:22.171727 22591 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34211 } }
I20260806 15:54:22.172010 22591 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:22.172093 22591 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2cdfe0c71e405db8375ead3af2d9c1, State: Initialized, Role: FOLLOWER
I20260806 15:54:22.172562 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34211 } }
I20260806 15:54:22.172699 22591 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:22.172746 22591 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:22.172817 22591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:22.173694 22591 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34211 } }
I20260806 15:54:22.174033 22591 leader_election.cc:302] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2cdfe0c71e405db8375ead3af2d9c1; no voters:
I20260806 15:54:22.174371 22591 leader_election.cc:288] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:22.174455 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:22.174639 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [term 1 LEADER]: Becoming Leader. State: Replica: cd2cdfe0c71e405db8375ead3af2d9c1, State: Running, Role: LEADER
I20260806 15:54:22.174942 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34211 } }
I20260806 15:54:22.175661 22591 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:22.176833 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34211 } } }
I20260806 15:54:22.176890 22606 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2cdfe0c71e405db8375ead3af2d9c1. Latest consensus state: current_term: 1 leader_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cdfe0c71e405db8375ead3af2d9c1" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34211 } } }
I20260806 15:54:22.176977 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:22.177009 22606 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:22.177574 22618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:22.180493 22618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:22.186544 22618 catalog_manager.cc:1347] Generated new cluster ID: de972118f9ae4d3c8f5b22685671b2ca
I20260806 15:54:22.186679 22618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:22.208277 22618 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:22.209682 22618 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:22.217316 22618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2cdfe0c71e405db8375ead3af2d9c1: Generated new TSK 0
I20260806 15:54:22.218088 22618 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 15:54:22.247630 22637 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 15:54:22.247740 22637 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 15:54:22.248375 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:22.232814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.13.223.62:34211)
0806 15:54:22.232959 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:22.232964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:22.233133 (+ 169us) server_negotiation.cc:407] Connection header received
0806 15:54:22.233262 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:22.233265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:22.233322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:22.233439 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0806 15:54:22.233680 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.234516 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.235412 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.235793 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.235896 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.236212 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 15:54:22.247975 (+ 11763us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 15:54:22.248241 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 15:54:22.248442 22597 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 15:54:22.248500 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:22.232751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34211 (local address 127.0.0.1:46920)
0806 15:54:22.233039 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:22.233060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:22.233157 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:22.233442 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:22.233444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:22.233450 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0806 15:54:22.233539 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:22.233579 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.234646 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.234649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:22.235295 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:22.235303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.235922 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.235925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:22.235996 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.236027 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 15:54:22.248170 (+ 12143us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 15:54:22.248320 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.62:34211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
I20260806 15:54:22.249330 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 22487
2026-08-06T15:54:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 15:54:22.295228 14204 mini_oidc.cc:152] Starting JWKS server
I20260806 15:54:22.296049 14204 webserver.cc:460] Webserver started at https://127.0.0.1:45347/ using document root <none> and password file <none>
I20260806 15:54:22.296140 14204 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 15:54:22.296394 14204 webserver.cc:460] Webserver started at http://127.0.0.1:37181/ using document root <none> and password file <none>
2026-08-06T15:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:22Z Disabled control of system clock
I20260806 15:54:22.306639 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:46627
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:46627
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45347/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 15:54:22.421461 22647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:22.421736 22647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:22.421774 22647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:22.426641 22647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:22.426745 22647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:22.426765 22647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:22.426815 22647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:22.426842 22647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 15:54:22.426867 22647 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 15:54:22.426893 22647 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45347/jwks.json
I20260806 15:54:22.433771 22647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:46627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45347/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:22.434969 22647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:22.436713 22647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:22.445652 22654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:22.445658 22657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:22.445673 22655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:22.445858 22647 server_base.cc:1034] running on GCE node
I20260806 15:54:22.446532 22647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:22.447664 22647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:22.448848 22647 hybrid_clock.cc:648] HybridClock initialized: now 1786031662448826 us; error 30 us; skew 500 ppm
I20260806 15:54:22.450742 22647 webserver.cc:460] Webserver started at http://127.13.223.62:43013/ using document root <none> and password file <none>
I20260806 15:54:22.451349 22647 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:22.451411 22647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:22.451624 22647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:22.453739 22647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "4b7fc8a4cf124697b988c978e922ad8a"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "05344c3aad7692ad3f5e8dcf8594d4fd"
server_key_iv: "2c3cd704c28a43b4e29baa80f2eeb7b6"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.454298 22647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "4b7fc8a4cf124697b988c978e922ad8a"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "05344c3aad7692ad3f5e8dcf8594d4fd"
server_key_iv: "2c3cd704c28a43b4e29baa80f2eeb7b6"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.457944 22647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:54:22.460418 22665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:22.461714 22647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:22.461870 22647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "4b7fc8a4cf124697b988c978e922ad8a"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "05344c3aad7692ad3f5e8dcf8594d4fd"
server_key_iv: "2c3cd704c28a43b4e29baa80f2eeb7b6"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.461997 22647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:22.485630 22647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:22.486361 22647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:22.486570 22647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:22.493669 22647 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:46627
I20260806 15:54:22.493678 22744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:46627 every 8 connection(s)
I20260806 15:54:22.494602 22647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:22.495651 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 22647
I20260806 15:54:22.495786 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:22.496124 14204 external_mini_cluster.cc:1428] Setting key 2f1e6610875cb8871574a7e5afbefed7
I20260806 15:54:22.498639 22745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:22.505746 22745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a: Bootstrap starting.
I20260806 15:54:22.506924 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:22.497648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.13.223.62:46627)
0806 15:54:22.498374 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:22.498399 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:22.498442 (+ 43us) server_negotiation.cc:407] Connection header received
0806 15:54:22.499316 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:22.499339 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:22.499821 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:22.500194 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:22.500653 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.502074 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.503066 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.503758 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.503919 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.504856 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:22.504862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:22.504877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:22.505383 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:22.505470 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:22.505496 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:22.505597 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:22.505720 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 15:54:22.505982 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":145,"threads_started":1}
I20260806 15:54:22.508338 22745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:22.509269 22745 log.cc:824] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:22.510017 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:54:22.511703 22745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a: No bootstrap required, opened a new log
I20260806 15:54:22.515650 22745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7fc8a4cf124697b988c978e922ad8a" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46627 } }
I20260806 15:54:22.515945 22745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:22.516028 22745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7fc8a4cf124697b988c978e922ad8a, State: Initialized, Role: FOLLOWER
I20260806 15:54:22.516443 22745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7fc8a4cf124697b988c978e922ad8a" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46627 } }
I20260806 15:54:22.516635 22745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:22.516722 22745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:22.516840 22745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:22.517781 22745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7fc8a4cf124697b988c978e922ad8a" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46627 } }
I20260806 15:54:22.518157 22745 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7fc8a4cf124697b988c978e922ad8a; no voters:
I20260806 15:54:22.518476 22745 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:22.518651 22750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:22.518930 22750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7fc8a4cf124697b988c978e922ad8a, State: Running, Role: LEADER
I20260806 15:54:22.519240 22750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7fc8a4cf124697b988c978e922ad8a" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46627 } }
I20260806 15:54:22.519908 22745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:22.521034 22754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7fc8a4cf124697b988c978e922ad8a. Latest consensus state: current_term: 1 leader_uuid: "4b7fc8a4cf124697b988c978e922ad8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7fc8a4cf124697b988c978e922ad8a" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46627 } } }
I20260806 15:54:22.521180 22754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:22.521576 22768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:22.521920 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7fc8a4cf124697b988c978e922ad8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7fc8a4cf124697b988c978e922ad8a" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46627 } } }
I20260806 15:54:22.522043 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:22.524773 22768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:22.529982 22768 catalog_manager.cc:1347] Generated new cluster ID: e159855a21f145a194022d50ebde462e
I20260806 15:54:22.530055 22768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:22.542650 22768 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:22.543565 22768 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:22.562728 22768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7fc8a4cf124697b988c978e922ad8a: Generated new TSK 0
I20260806 15:54:22.563580 22768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:22.583771 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:22.572938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.13.223.62:46627)
0806 15:54:22.573097 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:22.573101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:22.573248 (+ 147us) server_negotiation.cc:407] Connection header received
0806 15:54:22.573398 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:22.573402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:22.573460 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:22.573589 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0806 15:54:22.573805 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.574633 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.575590 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.575923 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.576027 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.576370 (+ 343us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 15:54:22.583340 (+ 6970us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 15:54:22.583621 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39122: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 15:54:22.583853 22748 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39122: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 15:54:22.586010 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:22.572811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:46627 (local address 127.0.0.1:39122)
0806 15:54:22.573157 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:22.573170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:22.573264 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:22.573582 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:22.573584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:22.573590 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0806 15:54:22.573694 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:22.573698 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.574798 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.574801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:22.575456 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:22.575464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.576079 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.576082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:22.576152 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.576176 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 15:54:22.583563 (+ 7387us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 15:54:22.585818 (+ 2255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.62:46627: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":1998,"thread_start_us":109,"threads_started":1}
I20260806 15:54:22.586707 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 22647
E20260806 15:54:22.606434 22793 webserver.cc:533] Webserver: sslize failed:
E20260806 15:54:22.606554 22793 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T15:54:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T15:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:22Z Disabled control of system clock
I20260806 15:54:22.647079 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:36019
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:36019
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/testchainca.cert with env {}
W20260806 15:54:22.758832 22797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:22.759078 22797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:22.759135 22797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:22.763690 22797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:22.763746 22797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:22.763762 22797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:22.763778 22797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:22.763794 22797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 15:54:22.763809 22797 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 15:54:22.763829 22797 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 15:54:22.768983 22797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:36019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:36019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:22.769968 22797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:22.770927 22797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:22.779773 22813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:22.779835 22810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:22.779933 22797 server_base.cc:1034] running on GCE node
W20260806 15:54:22.779835 22809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:22.780710 22797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:22.781787 22797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:22.782971 22797 hybrid_clock.cc:648] HybridClock initialized: now 1786031662782943 us; error 46 us; skew 500 ppm
I20260806 15:54:22.785022 22797 webserver.cc:460] Webserver started at http://127.13.223.62:34193/ using document root <none> and password file <none>
I20260806 15:54:22.785637 22797 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:22.785703 22797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:22.785917 22797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:22.787976 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "1b915604876342c4b8b23aa11a8c9914"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "f194a084f8ca4d0a27be4c78e2ad8a1f"
server_key_iv: "c76e4b760e700a5efb9261f2a6872f3d"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.788625 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "1b915604876342c4b8b23aa11a8c9914"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "f194a084f8ca4d0a27be4c78e2ad8a1f"
server_key_iv: "c76e4b760e700a5efb9261f2a6872f3d"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.792484 22797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 15:54:22.795128 22823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:22.796646 22797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 15:54:22.796826 22797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "1b915604876342c4b8b23aa11a8c9914"
format_stamp: "Formatted at 2026-08-06 15:54:22 on dist-test-slave-rwrg"
server_key: "f194a084f8ca4d0a27be4c78e2ad8a1f"
server_key_iv: "c76e4b760e700a5efb9261f2a6872f3d"
server_key_version: "encryptionkey@0"
I20260806 15:54:22.796952 22797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:22.814260 22797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:22.814965 22797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:22.815169 22797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:22.822642 22797 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:36019
I20260806 15:54:22.822674 22887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:36019 every 8 connection(s)
I20260806 15:54:22.823714 22797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:22.826646 22888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:22.832137 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914: Bootstrap starting.
I20260806 15:54:22.833338 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 22797
I20260806 15:54:22.833473 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:22.833899 14204 external_mini_cluster.cc:1428] Setting key dbbe8aaed2e067200d946652c887a035
I20260806 15:54:22.835095 22888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:22.835968 22888 log.cc:824] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:22.838032 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914: No bootstrap required, opened a new log
I20260806 15:54:22.841877 22888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b915604876342c4b8b23aa11a8c9914" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36019 } }
I20260806 15:54:22.842100 22888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:22.842175 22888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b915604876342c4b8b23aa11a8c9914, State: Initialized, Role: FOLLOWER
I20260806 15:54:22.842741 22888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b915604876342c4b8b23aa11a8c9914" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36019 } }
I20260806 15:54:22.842911 22888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:22.842984 22888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:22.843086 22888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:22.843652 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:22.835106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.13.223.62:36019)
0806 15:54:22.835823 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:22.835838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:22.835892 (+ 54us) server_negotiation.cc:407] Connection header received
0806 15:54:22.836749 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:22.836776 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:22.837197 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:22.837548 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:22.837975 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.838998 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.839957 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.840437 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.840697 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.841638 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:22.841643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:22.841657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:22.842048 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:22.842142 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:22.842178 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:22.842280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:22.842448 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 15:54:22.842723 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":151,"threads_started":1}
I20260806 15:54:22.844236 22888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b915604876342c4b8b23aa11a8c9914" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36019 } }
I20260806 15:54:22.844628 22888 leader_election.cc:302] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b915604876342c4b8b23aa11a8c9914; no voters:
I20260806 15:54:22.844995 22888 leader_election.cc:288] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:22.845067 22898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:22.845268 22898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [term 1 LEADER]: Becoming Leader. State: Replica: 1b915604876342c4b8b23aa11a8c9914, State: Running, Role: LEADER
I20260806 15:54:22.845551 22898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b915604876342c4b8b23aa11a8c9914" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36019 } }
I20260806 15:54:22.846306 22888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:22.846791 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:54:22.848336 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b915604876342c4b8b23aa11a8c9914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b915604876342c4b8b23aa11a8c9914" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36019 } } }
I20260806 15:54:22.848459 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:22.848433 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b915604876342c4b8b23aa11a8c9914. Latest consensus state: current_term: 1 leader_uuid: "1b915604876342c4b8b23aa11a8c9914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b915604876342c4b8b23aa11a8c9914" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36019 } } }
I20260806 15:54:22.848551 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914 [sys.catalog]: This master's current role is: LEADER
W20260806 15:54:22.851342 22919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:54:22.851423 22919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:54:22.851493 22915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:22.852283 22915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:22.857854 22915 catalog_manager.cc:1347] Generated new cluster ID: 39fff1fe5cf14674a1d4919d6b296819
I20260806 15:54:22.857939 22915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:22.875454 22915 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:22.876345 22915 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:22.880607 22915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b915604876342c4b8b23aa11a8c9914: Generated new TSK 0
I20260806 15:54:22.881352 22915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:23.381534 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:22.894144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.13.223.62:36019)
0806 15:54:22.894292 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:22.894296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:22.894481 (+ 185us) server_negotiation.cc:407] Connection header received
0806 15:54:22.894727 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:22.894730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:22.894778 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:22.894889 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0806 15:54:22.895157 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.895712 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.896628 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.896858 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.896969 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.897315 (+ 346us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 15:54:23.381013 (+483698us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 15:54:23.381372 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 15:54:23.381637 22893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 15:54:23.381953 22933 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:22.894073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:36019 (local address 127.0.0.1:45128)
0806 15:54:22.894391 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:22.894403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:22.894532 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:22.894899 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:22.894902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:22.894910 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 15:54:22.895043 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:22.895048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.895832 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.895836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:22.896462 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:22.896469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:22.896985 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:22.896989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:22.897096 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:22.897120 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 15:54:23.381286 (+484166us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 15:54:23.381484 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.62:36019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260806 15:54:23.382642 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 22797
2026-08-06T15:54:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (770 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:23 dist-test-slave-rwrg krb5kdc[22953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:23 dist-test-slave-rwrg krb5kdc[22953](info): set up 2 sockets
Aug 06 15:54:23 dist-test-slave-rwrg krb5kdc[22953](info): commencing operation
krb5kdc: starting...
W20260806 15:54:24.915723 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:25.044323 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40813
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:25.159080 22971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:25.159341 22971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:25.159381 22971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:25.163338 22971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:25.163391 22971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:25.163410 22971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:25.163427 22971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:25.163442 22971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:25.168751 22971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:25.169763 22971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:25.170704 22971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:25.179664 22981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.179680 22976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.179757 22978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:25.179839 22971 server_base.cc:1034] running on GCE node
I20260806 15:54:25.180649 22971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:25.181805 22971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:25.183012 22971 hybrid_clock.cc:648] HybridClock initialized: now 1786031665182981 us; error 42 us; skew 500 ppm
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:25.189321 22971 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:25.190454 22971 webserver.cc:460] Webserver started at http://127.13.223.62:46369/ using document root <none> and password file <none>
I20260806 15:54:25.191097 22971 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:25.191206 22971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:25.191483 22971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:25.193893 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "f30c1bd9abe5456ab5cf0263dba2b926"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "a4d7ba55e39f4e9eb04c82e78c6274d5"
server_key_iv: "67cc6554d9fdaabcc8e7ce19af8dc534"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.194568 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "f30c1bd9abe5456ab5cf0263dba2b926"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "a4d7ba55e39f4e9eb04c82e78c6274d5"
server_key_iv: "67cc6554d9fdaabcc8e7ce19af8dc534"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.198892 22971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 15:54:25.201782 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.203166 22971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:25.203361 22971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "f30c1bd9abe5456ab5cf0263dba2b926"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "a4d7ba55e39f4e9eb04c82e78c6274d5"
server_key_iv: "67cc6554d9fdaabcc8e7ce19af8dc534"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.203572 22971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:25.231943 22971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:25.235909 22971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:25.236162 22971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:25.244444 22971 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:40813
I20260806 15:54:25.244450 23061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:40813 every 8 connection(s)
I20260806 15:54:25.245633 22971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:25.248481 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:25.251837 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 22971
I20260806 15:54:25.252002 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:25.252271 14204 external_mini_cluster.cc:1428] Setting key 8efd907fc9b564b49a66a8cda6485eff
I20260806 15:54:25.254616 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926: Bootstrap starting.
I20260806 15:54:25.257198 23063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:25.257930 23063 log.cc:824] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:25.259826 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926: No bootstrap required, opened a new log
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:25.263947 23063 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40813 } }
I20260806 15:54:25.264166 23063 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:25.264225 23063 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30c1bd9abe5456ab5cf0263dba2b926, State: Initialized, Role: FOLLOWER
I20260806 15:54:25.264740 23063 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40813 } }
I20260806 15:54:25.264904 23063 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:25.264994 23063 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:25.265123 23063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:25.265967 23063 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40813 } }
I20260806 15:54:25.266331 23063 leader_election.cc:302] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30c1bd9abe5456ab5cf0263dba2b926; no voters:
I20260806 15:54:25.266666 23063 leader_election.cc:288] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:25.266803 23068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:25.267042 23068 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [term 1 LEADER]: Becoming Leader. State: Replica: f30c1bd9abe5456ab5cf0263dba2b926, State: Running, Role: LEADER
I20260806 15:54:25.267405 23068 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40813 } }
I20260806 15:54:25.267639 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:25.253733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.13.223.62:40813)
0806 15:54:25.254211 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:25.254231 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:25.254466 (+ 235us) server_negotiation.cc:407] Connection header received
0806 15:54:25.255294 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:25.255312 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:25.255644 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:25.256012 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:25.256932 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.258176 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.259036 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.259518 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.259694 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:25.262510 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:25.262516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:25.262537 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:25.262566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:25.264333 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:25.264852 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:25.264858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:25.264864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:25.265048 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:25.265352 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:25.265355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:25.265357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:25.265744 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:25.265970 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:25.266234 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 15:54:25.266575 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
I20260806 15:54:25.267970 23063 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:25.269908 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30c1bd9abe5456ab5cf0263dba2b926. Latest consensus state: current_term: 1 leader_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40813 } } }
I20260806 15:54:25.270121 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:25.270293 23070 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30c1bd9abe5456ab5cf0263dba2b926" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40813 } } }
I20260806 15:54:25.270658 23070 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926 [sys.catalog]: This master's current role is: LEADER
W20260806 15:54:25.273164 23093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:54:25.273253 23093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:54:25.273326 23095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:25.274068 23095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:25.280138 23095 catalog_manager.cc:1347] Generated new cluster ID: dabc63d5dfad4ec9b5e9c0b68f57a9dc
I20260806 15:54:25.280212 23095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:25.289597 23095 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:25.290812 23095 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:25.306454 23095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30c1bd9abe5456ab5cf0263dba2b926: Generated new TSK 0
I20260806 15:54:25.307221 23095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:25.341887 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40813
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:25.460449 23105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:25.460736 23105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:25.460781 23105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:25.464949 23105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:25.465009 23105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:25.465068 23105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:25.470592 23105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:25.471593 23105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:25.472572 23105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:25.482264 23111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.482296 23113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.482296 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:25.482412 23105 server_base.cc:1034] running on GCE node
I20260806 15:54:25.483031 23105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:25.483767 23105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:25.484983 23105 hybrid_clock.cc:648] HybridClock initialized: now 1786031665484968 us; error 26 us; skew 500 ppm
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:25.487896 23105 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:25.488999 23105 webserver.cc:460] Webserver started at http://127.13.223.1:44891/ using document root <none> and password file <none>
I20260806 15:54:25.489624 23105 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:25.489728 23105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:25.489984 23105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:25.492208 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "ba1f13bddc864962ad21a99ed132969c"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "4dfda6b82fd0723e5874df63fcadce9a"
server_key_iv: "fd3e189af23f04b16ce4a1d4b58a4444"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.492882 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "ba1f13bddc864962ad21a99ed132969c"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "4dfda6b82fd0723e5874df63fcadce9a"
server_key_iv: "fd3e189af23f04b16ce4a1d4b58a4444"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.497057 23105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:25.499794 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.501111 23105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:25.501255 23105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "ba1f13bddc864962ad21a99ed132969c"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "4dfda6b82fd0723e5874df63fcadce9a"
server_key_iv: "fd3e189af23f04b16ce4a1d4b58a4444"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.501374 23105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:25.514289 23105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:25.517238 23105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:25.517442 23105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:25.518033 23105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:25.518869 23105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:25.518939 23105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.518999 23105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:25.519055 23105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.528434 23105 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:40201
I20260806 15:54:25.528522 23299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:40201 every 8 connection(s)
I20260806 15:54:25.529369 23105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:25.539539 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 23105
I20260806 15:54:25.539772 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:25.540076 14204 external_mini_cluster.cc:1428] Setting key 67d78c9205fa5814725ef549d687e4b0
I20260806 15:54:25.542989 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:25.531176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:34001 (local address 127.13.223.62:40813)
0806 15:54:25.531332 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:25.531337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:25.532171 (+ 834us) server_negotiation.cc:407] Connection header received
0806 15:54:25.533563 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:25.533566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:25.533614 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:25.533688 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:25.534778 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.535351 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.536403 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.536701 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.536782 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:25.539339 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:25.539342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:25.539344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:25.539366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:25.540903 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:25.541487 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:25.541490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:25.541492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:25.541576 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:25.541918 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:25.541922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:25.541924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:25.542112 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:25.542221 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:25.542703 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 15:54:25.542826 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 15:54:25.543776 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:25.531577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40813 (local address 127.13.223.1:34001)
0806 15:54:25.532015 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:25.532058 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:25.533364 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:25.533817 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:25.533823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:25.534155 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:25.534638 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:25.534652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.535470 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.535472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:25.536288 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:25.536297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.536831 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.536834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:25.536970 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:25.537636 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:25.537651 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:25.539147 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:25.541010 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:25.541016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:25.541031 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:25.541371 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:25.541670 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:25.541673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:25.541676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:25.541819 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:25.542220 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:25.542228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:25.542468 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 15:54:25.542662 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 15:54:25.542964 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260806 15:54:25.545040 23300 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40813
I20260806 15:54:25.545296 23300 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:25.545935 23300 heartbeater.cc:511] Master 127.13.223.62:40813 requested a full tablet report, sending...
I20260806 15:54:25.547561 23013 ts_manager.cc:194] Registered new tserver with Master: ba1f13bddc864962ad21a99ed132969c (127.13.223.1:40201)
I20260806 15:54:25.548765 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:34001
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:25.571602 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40813
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:25.688418 23310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:25.688743 23310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:25.688818 23310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:25.692948 23310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:25.693001 23310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:25.693090 23310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:25.698446 23310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:25.699539 23310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:25.700657 23310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:25.712316 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.712311 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:25.712926 23310 server_base.cc:1034] running on GCE node
W20260806 15:54:25.713016 23323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:25.713389 23310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:25.714043 23310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:25.715243 23310 hybrid_clock.cc:648] HybridClock initialized: now 1786031665715219 us; error 33 us; skew 500 ppm
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:25.718451 23310 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:25.719326 23310 webserver.cc:460] Webserver started at http://127.13.223.2:43601/ using document root <none> and password file <none>
I20260806 15:54:25.719899 23310 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:25.720003 23310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:25.720243 23310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:25.722389 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "31967f8b68394ff2a27c3877dd803011"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "2e31bdf4afb5ff157cfd9de476170c53"
server_key_iv: "490f0e47635f4843335c60ee20e45a8d"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.722982 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "31967f8b68394ff2a27c3877dd803011"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "2e31bdf4afb5ff157cfd9de476170c53"
server_key_iv: "490f0e47635f4843335c60ee20e45a8d"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.727063 23310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:25.730103 23332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.731499 23310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:54:25.731654 23310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "31967f8b68394ff2a27c3877dd803011"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "2e31bdf4afb5ff157cfd9de476170c53"
server_key_iv: "490f0e47635f4843335c60ee20e45a8d"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.731786 23310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:25.743799 23310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:25.746797 23310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:25.747004 23310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:25.747635 23310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:25.748545 23310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:25.748624 23310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.748697 23310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:25.748755 23310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.757853 23310 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:39097
I20260806 15:54:25.757882 23509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:39097 every 8 connection(s)
I20260806 15:54:25.758776 23310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:25.759116 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 23310
I20260806 15:54:25.759248 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:25.759562 14204 external_mini_cluster.cc:1428] Setting key 041b97de859fd53f56d7b7ce5c3d2679
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:25.775556 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:25.760841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:38375 (local address 127.13.223.62:40813)
0806 15:54:25.761020 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:25.761024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:25.762086 (+ 1062us) server_negotiation.cc:407] Connection header received
0806 15:54:25.763894 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:25.763896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:25.763942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:25.764014 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:25.765655 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.766200 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.767704 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.767939 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.768065 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:25.771552 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:25.771555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:25.771557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:25.771584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:25.773222 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:25.773814 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:25.773817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:25.773818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:25.773898 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:25.774271 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:25.774275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:25.774276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:25.774503 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:25.774617 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:25.775222 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 15:54:25.775430 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:54:25.776610 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:25.761344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40813 (local address 127.13.223.2:38375)
0806 15:54:25.761906 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:25.761955 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:25.763633 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:25.764234 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:25.764247 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:25.764915 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:25.765492 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:25.765508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.766348 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.766354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:25.767535 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:25.767546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:25.768127 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:25.768132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:25.768309 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:25.769177 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:25.769195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:25.771348 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:25.773343 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:25.773349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:25.773363 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:25.773714 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:25.773988 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:25.773991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:25.773992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:25.774146 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:25.774657 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:25.774662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:25.774922 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 15:54:25.775114 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 15:54:25.775546 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":163,"threads_started":1}
I20260806 15:54:25.778065 23510 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40813
I20260806 15:54:25.778360 23510 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:25.779084 23510 heartbeater.cc:511] Master 127.13.223.62:40813 requested a full tablet report, sending...
I20260806 15:54:25.780386 23013 ts_manager.cc:194] Registered new tserver with Master: 31967f8b68394ff2a27c3877dd803011 (127.13.223.2:39097)
I20260806 15:54:25.781073 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:38375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:25.799520 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40813
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:25.916417 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:25.916772 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:25.916846 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:25.920851 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:25.920943 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:25.921032 23520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:25.926175 23520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:25.927240 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:25.928257 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:25.938764 23527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.938766 23526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:25.938742 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:25.939222 23520 server_base.cc:1034] running on GCE node
I20260806 15:54:25.939690 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:25.940408 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:25.941654 23520 hybrid_clock.cc:648] HybridClock initialized: now 1786031665941621 us; error 58 us; skew 500 ppm
Aug 06 15:54:25 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:25.944682 23520 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:25.945603 23520 webserver.cc:460] Webserver started at http://127.13.223.3:44879/ using document root <none> and password file <none>
I20260806 15:54:25.946269 23520 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:25.946369 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:25.946626 23520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:25.948917 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "5a483da1057a4847aac085a30ecb3a1a"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "7ba53acb788ef8f35c116c2118acdefd"
server_key_iv: "e0a4f20602ce24ef6bbbadd70a22e516"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.949565 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "5a483da1057a4847aac085a30ecb3a1a"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "7ba53acb788ef8f35c116c2118acdefd"
server_key_iv: "e0a4f20602ce24ef6bbbadd70a22e516"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.953625 23520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:25.956275 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:25.957825 23520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:54:25.957935 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "5a483da1057a4847aac085a30ecb3a1a"
format_stamp: "Formatted at 2026-08-06 15:54:25 on dist-test-slave-rwrg"
server_key: "7ba53acb788ef8f35c116c2118acdefd"
server_key_iv: "e0a4f20602ce24ef6bbbadd70a22e516"
server_key_version: "encryptionkey@0"
I20260806 15:54:25.958034 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:25.998718 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:26.002535 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:26.002717 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:26.003317 23520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:26.004359 23520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:26.004411 23520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:26.004489 23520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:26.004565 23520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 15:54:26.014716 23520 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:44767
I20260806 15:54:26.014746 23714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:44767 every 8 connection(s)
I20260806 15:54:26.015739 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:26.017622 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 23520
I20260806 15:54:26.017750 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:26.018106 14204 external_mini_cluster.cc:1428] Setting key 518f10e152a4d2d9763b460b3286f4d7
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031665, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:26.030624 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:26.017709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:43327 (local address 127.13.223.62:40813)
0806 15:54:26.017897 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.017903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.019036 (+ 1133us) server_negotiation.cc:407] Connection header received
0806 15:54:26.020880 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.020882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.020927 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.020999 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.022260 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.022803 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.024012 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.024247 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.024329 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:26.026971 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:26.026974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:26.026976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:26.027001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:26.028347 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.028898 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.028900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.028902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.028980 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.029297 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.029299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.029300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.029557 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:26.029835 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:26.030190 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 15:54:26.030429 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 15:54:26.031446 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:26.018247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40813 (local address 127.13.223.3:43327)
0806 15:54:26.018860 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:26.018926 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:26.020686 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:26.021170 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:26.021177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:26.021613 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:26.022123 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:26.022138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.022924 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.022927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:26.023886 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:26.023897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.024388 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.024391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:26.024658 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:26.025267 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:26.025284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:26.026812 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:26.028470 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:26.028475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:26.028541 (+ 66us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:26.028807 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:26.029064 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:26.029066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:26.029068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:26.029202 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:26.029690 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:26.029696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:26.029935 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 15:54:26.030112 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 15:54:26.030374 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":145,"threads_started":1}
I20260806 15:54:26.032827 23717 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40813
I20260806 15:54:26.033082 23717 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:26.033614 23717 heartbeater.cc:511] Master 127.13.223.62:40813 requested a full tablet report, sending...
I20260806 15:54:26.034732 23011 ts_manager.cc:194] Registered new tserver with Master: 5a483da1057a4847aac085a30ecb3a1a (127.13.223.3:44767)
I20260806 15:54:26.035346 23011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:43327
I20260806 15:54:26.044289 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/test-admin.keytab.
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.061381 14204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 15:54:26.064610 23728 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:26.062831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.13.223.1:40201)
0806 15:54:26.063276 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.063301 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.063323 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:54:26.063372 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.063378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.063574 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.063743 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.064383 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39798: BlockingRecv error: recv got EOF from 127.0.0.1:39798 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
W20260806 15:54:26.064718 14204 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc' of type FILE
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.065727 14204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.066615 14204 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:54:26.076359 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:26.067479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.13.223.1:40201)
0806 15:54:26.067608 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.067615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.067696 (+ 81us) server_negotiation.cc:407] Connection header received
0806 15:54:26.067835 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.067838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.067883 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.067965 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.068482 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.069332 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.070175 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.070634 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.070748 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:26.072978 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:26.072988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:26.073006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:26.073035 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:26.074506 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.074970 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.074983 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.074985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.075061 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.075348 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.075352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.075354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.075620 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:26.075832 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:26.076069 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 15:54:26.076232 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 15:54:26.080282 23728 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:26.079270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.13.223.1:40201)
0806 15:54:26.079397 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.079401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.079518 (+ 117us) server_negotiation.cc:407] Connection header received
0806 15:54:26.079659 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.079662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.079701 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.079791 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.080179 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39814: BlockingRecv error: recv got EOF from 127.0.0.1:39814 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 15:54:26.080422 14204 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc' of type FILE
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.081485 14204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.082438 14204 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:54:26.091027 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:26.083290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.13.223.1:40201)
0806 15:54:26.083433 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.083436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.083449 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:54:26.083619 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.083621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.083664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.083743 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.084222 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.084835 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.085609 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.085821 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.085901 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:26.088210 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:26.088213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:26.088215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:26.088238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:26.089538 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.089925 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.089928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.089929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.090010 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.090269 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.090271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.090272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.090424 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:26.090537 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:26.090708 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 15:54:26.090844 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 15:54:26.093983 23728 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:26.092960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.13.223.1:40201)
0806 15:54:26.093106 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.093111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.093206 (+ 95us) server_negotiation.cc:407] Connection header received
0806 15:54:26.093351 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.093354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.093396 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.093474 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.093867 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39834: BlockingRecv error: recv got EOF from 127.0.0.1:39834 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 15:54:26.094169 14204 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786031632208048-14204-0/krb5kdc/krb5cc' of type FILE
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.095235 14204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:26.096143 14204 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:54:26.104748 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:26.096968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.13.223.1:40201)
0806 15:54:26.097120 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:26.097123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:26.097187 (+ 64us) server_negotiation.cc:407] Connection header received
0806 15:54:26.097257 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:26.097259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:26.097293 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:26.097358 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:26.097816 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.098336 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.099212 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:26.099422 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:26.099512 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:26.101961 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:26.101964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:26.101966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:26.101988 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:26.103278 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.103647 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.103650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.103651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.103722 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:26.103946 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:26.103948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:26.103949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:26.104182 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:26.104303 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:26.104479 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 15:54:26.104641 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 15:54:26.105852 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 23105
I20260806 15:54:26.120975 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 23310
I20260806 15:54:26.132193 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 23520
I20260806 15:54:26.141301 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 22971
2026-08-06T15:54:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2503 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[23752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[23752](info): set up 2 sockets
Aug 06 15:54:26 dist-test-slave-rwrg krb5kdc[23752](info): commencing operation
krb5kdc: starting...
W20260806 15:54:27.397202 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:27 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab.
Entry for principal oryx/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab.
I20260806 15:54:27.533809 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44803
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab
--principal=oryx/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:27.647897 23771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:27.648247 23771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:27.648324 23771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:27.652371 23771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:27.652462 23771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:27.652496 23771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:27.652601 23771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:27.652633 23771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:27.658107 23771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab
--principal=oryx/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:27.659220 23771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:27.660280 23771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:27.668977 23786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:27.669130 23782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:27.668956 23784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:27.669345 23771 server_base.cc:1034] running on GCE node
I20260806 15:54:27.670024 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:27.671176 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:27.672422 23771 hybrid_clock.cc:648] HybridClock initialized: now 1786031667672385 us; error 47 us; skew 500 ppm
Aug 06 15:54:27 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:27.675491 23771 init.cc:377] Logged in from keytab as oryx/127.13.223.62@KRBTEST.COM (short username oryx)
I20260806 15:54:27.676461 23771 webserver.cc:460] Webserver started at http://127.13.223.62:36991/ using document root <none> and password file <none>
I20260806 15:54:27.677090 23771 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:27.677191 23771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:27.677435 23771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:27.679533 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "38e98a7e487340bab964f5dfbbd35b70"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "dda95ec3900d1db5676f0921df25cedb"
server_key_iv: "8d15370f3f3aed04e7a1c51a55dc1f46"
server_key_version: "encryptionkey@0"
I20260806 15:54:27.680191 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "38e98a7e487340bab964f5dfbbd35b70"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "dda95ec3900d1db5676f0921df25cedb"
server_key_iv: "8d15370f3f3aed04e7a1c51a55dc1f46"
server_key_version: "encryptionkey@0"
I20260806 15:54:27.684026 23771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:27.686614 23798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:27.687952 23771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 15:54:27.688141 23771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "38e98a7e487340bab964f5dfbbd35b70"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "dda95ec3900d1db5676f0921df25cedb"
server_key_iv: "8d15370f3f3aed04e7a1c51a55dc1f46"
server_key_version: "encryptionkey@0"
I20260806 15:54:27.688267 23771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:27.730090 23771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:27.733527 23771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:27.733754 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:27.741451 23771 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:44803
I20260806 15:54:27.741763 23883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:44803 every 8 connection(s)
I20260806 15:54:27.742591 23771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:27.745426 23885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:27.750782 23885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Bootstrap starting.
I20260806 15:54:27.751103 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 23771
I20260806 15:54:27.751202 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:27.751422 14204 external_mini_cluster.cc:1428] Setting key f78374e9ba27379f4d45230bf50fe4f1
I20260806 15:54:27.753319 23885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:27.754110 23885 log.cc:824] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:27.756374 23885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: No bootstrap required, opened a new log
Aug 06 15:54:27 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:27.761102 23885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } }
I20260806 15:54:27.761353 23885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:27.761435 23885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e98a7e487340bab964f5dfbbd35b70, State: Initialized, Role: FOLLOWER
I20260806 15:54:27.761924 23885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } }
I20260806 15:54:27.762111 23885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:27.762197 23885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:27.762310 23885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:27.763356 23885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } }
I20260806 15:54:27.763695 23885 leader_election.cc:302] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38e98a7e487340bab964f5dfbbd35b70; no voters:
I20260806 15:54:27.764006 23885 leader_election.cc:288] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:27.764168 23891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:27.764375 23891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 LEADER]: Becoming Leader. State: Replica: 38e98a7e487340bab964f5dfbbd35b70, State: Running, Role: LEADER
I20260806 15:54:27.764746 23891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } }
I20260806 15:54:27.765373 23885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:27.765991 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e98a7e487340bab964f5dfbbd35b70. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } }
I20260806 15:54:27.766146 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:27.766525 23899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:27.766722 23893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } }
I20260806 15:54:27.766888 23893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:27.767383 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:27.752782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.13.223.62:44803)
0806 15:54:27.753222 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:27.753237 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:27.753270 (+ 33us) server_negotiation.cc:407] Connection header received
0806 15:54:27.753956 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:27.753992 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:27.754426 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:27.754843 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:27.755770 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:27.756836 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:27.758005 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:27.758453 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:27.758594 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:27.761701 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:27.761707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:27.761721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:27.761749 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:27.763648 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:27.764177 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:27.764185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:27.764194 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:27.764363 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:27.764739 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:27.764743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:27.764745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:27.765270 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:27.765479 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:27.765882 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 15:54:27.766402 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260806 15:54:27.769659 23899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:27.775002 23899 catalog_manager.cc:1347] Generated new cluster ID: e96f23a66721400d9de599be83617ee0
I20260806 15:54:27.775079 23899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:27.783207 23899 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:27.784052 23899 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:27.791766 23899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Generated new TSK 0
I20260806 15:54:27.792620 23899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab.
Entry for principal oryx/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab.
I20260806 15:54:27.829664 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44803
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:27.942790 23918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:27.943053 23918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:27.943096 23918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:27.947103 23918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:27.947160 23918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:27.947261 23918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:27.952525 23918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:27.953577 23918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:27.954649 23918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:54:27.964226 23918 server_base.cc:1034] running on GCE node
W20260806 15:54:27.964192 23928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:27.964202 23932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:27.964395 23929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:27.964808 23918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:27.965508 23918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:27.966683 23918 hybrid_clock.cc:648] HybridClock initialized: now 1786031667966660 us; error 32 us; skew 500 ppm
Aug 06 15:54:27 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:27.969756 23918 init.cc:377] Logged in from keytab as oryx/127.13.223.1@KRBTEST.COM (short username oryx)
I20260806 15:54:27.970590 23918 webserver.cc:460] Webserver started at http://127.13.223.1:41263/ using document root <none> and password file <none>
I20260806 15:54:27.971143 23918 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:27.971233 23918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:27.971463 23918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:27.973505 23918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "95c9f851accc49cd900e4b1c97970ace"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "f54dce5bfab111ed5e1bb9535f5fe6a9"
server_key_iv: "f631d87dc7f469679791b8d1e9dc6e17"
server_key_version: "encryptionkey@0"
I20260806 15:54:27.974072 23918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "95c9f851accc49cd900e4b1c97970ace"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "f54dce5bfab111ed5e1bb9535f5fe6a9"
server_key_iv: "f631d87dc7f469679791b8d1e9dc6e17"
server_key_version: "encryptionkey@0"
I20260806 15:54:27.977808 23918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:54:27.980195 23945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:27.981386 23918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:27.981519 23918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "95c9f851accc49cd900e4b1c97970ace"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "f54dce5bfab111ed5e1bb9535f5fe6a9"
server_key_iv: "f631d87dc7f469679791b8d1e9dc6e17"
server_key_version: "encryptionkey@0"
I20260806 15:54:27.981627 23918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:27.994180 23918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:27.996991 23918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:27.997192 23918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:27.997773 23918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:27.998581 23918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:27.998652 23918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:27.998726 23918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:27.998769 23918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.008272 23918 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:35767
I20260806 15:54:28.008313 24106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:35767 every 8 connection(s)
I20260806 15:54:28.009205 23918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:28.016708 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 23918
I20260806 15:54:28.016835 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:28.017169 14204 external_mini_cluster.cc:1428] Setting key df67e471d09b3bc7743193797575cc83
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:28.023741 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.011180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:60377 (local address 127.13.223.62:44803)
0806 15:54:28.011360 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.011365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.012152 (+ 787us) server_negotiation.cc:407] Connection header received
0806 15:54:28.013495 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.013498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.013546 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.013621 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.014795 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.015469 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.016702 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.017071 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.017192 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.019712 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.019719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.019726 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.019760 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.021385 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.022033 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.022035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.022037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.022113 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.022505 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.022507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.022508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.022679 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.022754 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.023402 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.023508 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 15:54:28.024886 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.011550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:60377)
0806 15:54:28.011998 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.012042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.013300 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.013802 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.013809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.014158 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.014639 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.014654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.015591 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.015593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.016481 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.016544 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.017237 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.017242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.017433 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.018066 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.018083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.019536 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.021504 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.021510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.021529 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.021931 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.022206 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.022212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.022217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.022407 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.022787 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.022796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.023116 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.023358 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.023732 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260806 15:54:28.026216 24107 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44803
I20260806 15:54:28.026458 24107 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:28.026937 24107 heartbeater.cc:511] Master 127.13.223.62:44803 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:54:28.028601 23823 ts_manager.cc:194] Registered new tserver with Master: 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767)
I20260806 15:54:28.029841 23823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:60377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab.
Entry for principal oryx/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab.
I20260806 15:54:28.046881 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44803
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:28.157831 24117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:28.158124 24117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:28.158217 24117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:28.162194 24117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:28.162271 24117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:28.162357 24117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:28.167837 24117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:28.168905 24117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:28.169932 24117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:28.179505 24130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:28.179648 24117 server_base.cc:1034] running on GCE node
W20260806 15:54:28.179567 24127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:28.179502 24124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:28.180177 24117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:28.180955 24117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:28.182164 24117 hybrid_clock.cc:648] HybridClock initialized: now 1786031668182155 us; error 30 us; skew 500 ppm
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:28.185240 24117 init.cc:377] Logged in from keytab as oryx/127.13.223.2@KRBTEST.COM (short username oryx)
I20260806 15:54:28.186287 24117 webserver.cc:460] Webserver started at http://127.13.223.2:42311/ using document root <none> and password file <none>
I20260806 15:54:28.186872 24117 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:28.186940 24117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:28.187196 24117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:28.189447 24117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "b418ae20483d4f17aab8b7620bd13f02"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "f81bc32d270a8327bd2a22194ea8e1cb"
server_key_iv: "abf439ab7ee5a019d2d70cc811df0d00"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.190055 24117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "b418ae20483d4f17aab8b7620bd13f02"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "f81bc32d270a8327bd2a22194ea8e1cb"
server_key_iv: "abf439ab7ee5a019d2d70cc811df0d00"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.194329 24117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:28.197059 24149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.198280 24117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:28.198426 24117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "b418ae20483d4f17aab8b7620bd13f02"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "f81bc32d270a8327bd2a22194ea8e1cb"
server_key_iv: "abf439ab7ee5a019d2d70cc811df0d00"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.198534 24117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:28.212056 24117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:28.215025 24117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:28.215238 24117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:28.215879 24117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:28.216826 24117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:28.216878 24117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.216953 24117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:28.216996 24117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.226575 24117 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:34903
I20260806 15:54:28.226598 24334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:34903 every 8 connection(s)
I20260806 15:54:28.227478 24117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:28.233762 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 24117
I20260806 15:54:28.233918 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:28.234328 14204 external_mini_cluster.cc:1428] Setting key d231e9070d20a90d970008336482cbe1
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:28.242481 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.229280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:38247 (local address 127.13.223.62:44803)
0806 15:54:28.229486 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.229491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.230361 (+ 870us) server_negotiation.cc:407] Connection header received
0806 15:54:28.231644 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.231648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.231699 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.231790 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.232975 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.233679 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.235057 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.235348 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.235440 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.238052 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.238056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.238076 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.238124 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.239860 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.240634 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.240637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.240638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.240721 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.241173 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.241176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.241177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.241398 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.241508 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.242233 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.242363 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 15:54:28.243853 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.229722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:38247)
0806 15:54:28.230198 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.230241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.231441 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.231933 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.231939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.232276 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.232840 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.232854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.233811 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.233813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.234944 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.234954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.235504 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.235508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.235707 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.236312 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.236328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.237796 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.239999 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.240005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.240021 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.240454 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.240829 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.240836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.240842 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.241049 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.241514 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.241524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.241908 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.242147 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.242588 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260806 15:54:28.245173 24335 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44803
I20260806 15:54:28.245416 24335 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:28.245904 24335 heartbeater.cc:511] Master 127.13.223.62:44803 requested a full tablet report, sending...
I20260806 15:54:28.247013 23823 ts_manager.cc:194] Registered new tserver with Master: b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903)
WARNING: no policy specified for oryx/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:28.247640 23823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:38247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab.
Entry for principal oryx/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab.
I20260806 15:54:28.267292 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44803
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:28.381774 24342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:28.382076 24342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:28.382153 24342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:28.386303 24342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:28.386386 24342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:28.386482 24342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:28.391847 24342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:28.393070 24342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:28.394138 24342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:28.404168 24349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:28.404181 24352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:28.404192 24348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:28.404443 24342 server_base.cc:1034] running on GCE node
I20260806 15:54:28.404958 24342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:28.405663 24342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:28.406869 24342 hybrid_clock.cc:648] HybridClock initialized: now 1786031668406857 us; error 33 us; skew 500 ppm
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:28.409963 24342 init.cc:377] Logged in from keytab as oryx/127.13.223.3@KRBTEST.COM (short username oryx)
I20260806 15:54:28.410856 24342 webserver.cc:460] Webserver started at http://127.13.223.3:32957/ using document root <none> and password file <none>
I20260806 15:54:28.411413 24342 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:28.411504 24342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:28.411758 24342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:28.413910 24342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "a74caae195ad48cabe9946b38b7afe6b"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "0f17e6aa167577c6078155b1c68074b0"
server_key_iv: "b19d7ba3c334142622dd6469b91eda67"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.414490 24342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "a74caae195ad48cabe9946b38b7afe6b"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "0f17e6aa167577c6078155b1c68074b0"
server_key_iv: "b19d7ba3c334142622dd6469b91eda67"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.418273 24342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:28.420856 24362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.422106 24342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:28.422240 24342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "a74caae195ad48cabe9946b38b7afe6b"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "0f17e6aa167577c6078155b1c68074b0"
server_key_iv: "b19d7ba3c334142622dd6469b91eda67"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.422348 24342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:28.454203 24342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:28.457726 24342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:28.457959 24342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:28.458595 24342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:28.459509 24342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:28.459584 24342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.459659 24342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:28.459704 24342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.469966 24342 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40601
I20260806 15:54:28.469990 24538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40601 every 8 connection(s)
I20260806 15:54:28.470867 24342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:28.474424 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 24342
I20260806 15:54:28.474525 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:28.474813 14204 external_mini_cluster.cc:1428] Setting key 253dcc803c5f5dec2dab7f9becaa5e9a
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:28.485580 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.472913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:47311 (local address 127.13.223.62:44803)
0806 15:54:28.473065 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.473069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.473937 (+ 868us) server_negotiation.cc:407] Connection header received
0806 15:54:28.475580 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.475583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.475631 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.475741 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.477409 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.477953 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.479175 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.479396 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.479470 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.482000 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.482003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.482005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.482030 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.483486 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.483933 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.483936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.483937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.484009 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.484306 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.484308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.484310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.484571 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.484686 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.485278 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.485401 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 15:54:28.486433 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.473290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:47311)
0806 15:54:28.473768 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.473812 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.475342 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.475963 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.475970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.476551 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.477242 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.477260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.478064 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.478067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.479069 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.479079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.479523 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.479526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.479663 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.480266 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.480282 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.481844 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.483584 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.483588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.483601 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.483849 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.484098 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.484100 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.484101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.484228 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.484685 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.484694 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.484953 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.485226 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.485579 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260806 15:54:28.487787 24539 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44803
I20260806 15:54:28.488085 24539 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:28.488749 24539 heartbeater.cc:511] Master 127.13.223.62:44803 requested a full tablet report, sending...
I20260806 15:54:28.489879 23823 ts_manager.cc:194] Registered new tserver with Master: a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601)
I20260806 15:54:28.490413 23823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:47311
I20260806 15:54:28.501235 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.223.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.61.keytab.
Entry for principal oryx/127.13.223.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.61.keytab.
I20260806 15:54:28.529532 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:39527
--webserver_interface=127.13.223.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.61.keytab
--principal=oryx/127.13.223.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44803,127.13.223.61:39527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:28.644317 24547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:28.644614 24547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:28.644657 24547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:28.648587 24547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:28.648640 24547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:28.648659 24547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:28.648675 24547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:28.648691 24547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:28.653901 24547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44803,127.13.223.61:39527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.61.keytab
--principal=oryx/127.13.223.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
W20260806 15:54:28.654090 24547 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.223.62:44803,127.13.223.61:39527'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 15:54:28.654985 24547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:28.655936 24547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:28.664062 24557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:28.664161 24560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:28.664171 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:28.664395 24547 server_base.cc:1034] running on GCE node
I20260806 15:54:28.665040 24547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:28.666188 24547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:28.667359 24547 hybrid_clock.cc:648] HybridClock initialized: now 1786031668667346 us; error 24 us; skew 500 ppm
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:28.670233 24547 init.cc:377] Logged in from keytab as oryx/127.13.223.61@KRBTEST.COM (short username oryx)
I20260806 15:54:28.671072 24547 webserver.cc:460] Webserver started at http://127.13.223.61:39781/ using document root <none> and password file <none>
I20260806 15:54:28.671656 24547 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:28.671749 24547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:28.671973 24547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:28.674067 24547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/instance:
uuid: "b59818fcaf734e5b85ce691bda0e822a"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "40e69efd62872e63413bb25e7f8006d6"
server_key_iv: "94b4b3a23e684cd495146cfd7e5154b8"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.674667 24547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal/instance:
uuid: "b59818fcaf734e5b85ce691bda0e822a"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "40e69efd62872e63413bb25e7f8006d6"
server_key_iv: "94b4b3a23e684cd495146cfd7e5154b8"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.678450 24547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:28.681059 24576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.682346 24547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:28.682567 24547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
uuid: "b59818fcaf734e5b85ce691bda0e822a"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "40e69efd62872e63413bb25e7f8006d6"
server_key_iv: "94b4b3a23e684cd495146cfd7e5154b8"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.682695 24547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:28.694446 24547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:28.697484 24547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:28.697696 24547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:28.705575 24547 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.61:39527
I20260806 15:54:28.705862 24656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.61:39527 every 8 connection(s)
I20260806 15:54:28.706595 24547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
I20260806 15:54:28.709535 24661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:28.713376 24661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:54:28.716769 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 24547
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.61@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
I20260806 15:54:28.729218 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.715243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.13.223.62:44803)
0806 15:54:28.715380 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.715383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.716268 (+ 885us) server_negotiation.cc:407] Connection header received
0806 15:54:28.717285 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.717292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.717365 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.717453 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.719079 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.719831 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.721292 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.721641 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.721752 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.724863 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.724868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.724871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.724924 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.726770 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.727292 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.727296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.727298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.727387 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.727742 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.727748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.727750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.728087 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.728212 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.728905 (+ 693us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.729037 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:54:28.730114 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.717851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.13.223.61:39527)
0806 15:54:28.718262 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.718274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.718301 (+ 27us) server_negotiation.cc:407] Connection header received
0806 15:54:28.718348 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.718353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.718534 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.718709 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.719658 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.720936 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.722052 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.722597 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.722731 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.726078 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.726084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.726094 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.726116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.727764 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.728201 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.728207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.728208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.728325 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.728658 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.728662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.728663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.728903 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.729130 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.729568 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.729700 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":317,"spinlock_wait_cycles":2176,"thread_start_us":224,"threads_started":1}
I20260806 15:54:28.730121 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.715644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52402)
0806 15:54:28.716110 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.716151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.717064 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.717772 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.717782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.718307 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.718911 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.718926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.719972 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.719976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.721158 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.721166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.721786 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.721788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.722048 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.722754 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.722785 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.724683 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.726872 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.726881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.726891 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.727181 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.727481 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.727485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.727487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.727638 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.728291 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.728297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.728647 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.728841 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.729175 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":66,"spinlock_wait_cycles":6656,"thread_start_us":129,"threads_started":1}
I20260806 15:54:28.731645 24661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.61@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
I20260806 15:54:28.739389 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.732147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45294)
0806 15:54:28.732233 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.732247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.732338 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.732619 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.732622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.732786 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.732857 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.732862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.733604 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.733607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.734100 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.734107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.734606 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.734609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.734714 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.735183 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.735204 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.736323 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.737955 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.737961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.737963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.738158 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.738416 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.738421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.738423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.738502 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.738866 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.738869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.738972 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.739080 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.739205 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260806 15:54:28.739389 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.732225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.13.223.61:39527)
0806 15:54:28.732362 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.732367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.732390 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:54:28.732423 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.732425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.732464 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.732608 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.732964 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.733468 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.734206 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.734469 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.734602 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.736445 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.736449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.736451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.736479 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.737851 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.738250 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.738256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.738258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.738330 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.738590 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.738594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.738596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.738773 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.738877 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.739119 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.739256 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":40}
I20260806 15:54:28.742818 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Bootstrap starting.
I20260806 15:54:28.745240 24661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:28.745957 24661 log.cc:824] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:28.747792 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: No bootstrap required, opened a new log
I20260806 15:54:28.750768 24661 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } }
I20260806 15:54:28.750957 24661 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:28.751027 24661 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59818fcaf734e5b85ce691bda0e822a, State: Initialized, Role: FOLLOWER
I20260806 15:54:28.751482 24661 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } }
I20260806 15:54:28.751955 24668 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } } }
I20260806 15:54:28.752120 24668 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:28.752729 24661 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: configured and running, proceeding with master startup.
W20260806 15:54:28.757442 24687 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:54:28.757530 24687 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:54:28.759688 24547 master_runner.cc:428] Detected that this master b59818fcaf734e5b85ce691bda0e822a is joining an existing cluster
I20260806 15:54:28.759781 24547 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.223.61:39527
I20260806 15:54:28.771396 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.762436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.13.223.61:39527)
0806 15:54:28.762550 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.762554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.762568 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:28.762685 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.762688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.762725 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.762798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.763369 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.764088 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.765477 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.765815 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.765935 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.767647 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.767652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.767656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.767684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.769596 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770090 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.770096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.770101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.770181 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770501 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.770504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.770506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.770727 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.770861 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.771119 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.771233 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260806 15:54:28.771780 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.762065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52406)
0806 15:54:28.762382 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.762400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.762555 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.763081 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.763085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.763316 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.763459 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.763467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.764452 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.764457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.765248 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.765259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.765930 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.765934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.766225 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.767379 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.767392 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.767918 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.770006 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.770012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.770275 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.770603 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.770608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.770700 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.771178 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.771182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.771286 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.771419 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.771569 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":104,"thread_start_us":112,"threads_started":1}
I20260806 15:54:28.771773 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.762119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.13.223.62:44803)
0806 15:54:28.762262 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.762269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.762474 (+ 205us) server_negotiation.cc:407] Connection header received
0806 15:54:28.762657 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.762661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.762714 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.762798 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.763592 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.764335 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.765387 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.765735 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.765861 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.768056 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.768060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.768063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.768091 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.769886 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770364 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.770369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.770371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.770488 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770807 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.770814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.770819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.771076 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.771189 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.771492 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.771624 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 15:54:28.771822 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.762155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45296)
0806 15:54:28.762484 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.762497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.762582 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.762907 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.762910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.763130 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.763246 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.763252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.764211 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.764215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.765324 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.765337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.766027 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.766030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.766251 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.766989 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.767005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.767483 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.769726 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.769729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.769732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.769993 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.770291 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.770294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.770296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.770394 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.770840 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.770843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.770941 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.771520 (+ 579us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.771648 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":131,"thread_start_us":107,"threads_started":1}
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.62@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
I20260806 15:54:28.786865 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.776881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.13.223.61:39527)
0806 15:54:28.776998 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.777002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.777339 (+ 337us) server_negotiation.cc:407] Connection header received
0806 15:54:28.777551 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.777554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.777608 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.777678 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.778271 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.779042 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.779912 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.780261 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.780366 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.782931 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.782934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.782937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.782967 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.784729 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.785323 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.785330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.785334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.785427 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.785739 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.785745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.785750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.786017 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.786120 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.786590 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.786707 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 15:54:28.787410 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.776830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45312)
0806 15:54:28.777203 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.777255 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.777396 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.777711 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.777715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.778051 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.778154 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.778160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.779167 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.779170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.779793 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.779802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.780402 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.780406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.780599 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.781102 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.781114 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.782789 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.784847 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.784857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.784864 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.785221 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.785532 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.785535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.785537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.785638 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.786136 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.786144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.786274 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.787005 (+ 731us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.787227 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260806 15:54:28.789014 23823 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.223.61:39527: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a"
member_type: NON_VOTER
last_known_addr {
host: "127.13.223.61"
port: 39527
}
attrs {
promote: true
}
}
dest_uuid: "38e98a7e487340bab964f5dfbbd35b70"
cas_config_opid_index: -1
I20260806 15:54:28.789714 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } }
I20260806 15:54:28.791430 24703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b59818fcaf734e5b85ce691bda0e822a (127.13.223.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } }
I20260806 15:54:28.792434 24613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:28.793188 24703 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.223.61:39527
I20260806 15:54:28.793663 24547 master.cc:537] Master@127.13.223.61:39527 shutting down...
I20260806 15:54:28.793661 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } }
I20260806 15:54:28.793759 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:28.794477 24613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Refusing update from remote peer 38e98a7e487340bab964f5dfbbd35b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 15:54:28.795224 24701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.13.223.60@KRBTEST.COM; defaulting to no policy
W20260806 15:54:28.798095 23804 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 15:54:28.798424 24547 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:28.798697 24547 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:28.798825 24547 tablet_replica.cc:330] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.60@KRBTEST.COM" created.
W20260806 15:54:28.799162 23804 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 -> Peer b59818fcaf734e5b85ce691bda0e822a (127.13.223.61:39527): Couldn't send request to peer b59818fcaf734e5b85ce691bda0e822a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 15:54:28.806464 24547 master.cc:552] Master@127.13.223.61:39527 shutdown complete.
I20260806 15:54:28.806542 24547 master_runner.cc:305] Clearing existing system tablet
I20260806 15:54:28.806818 24547 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:28.806888 24547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:28.808488 24547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 15:54:28.809638 24711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.809902 24547 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:28.810052 24547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 15:54:28.810101 24547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
uuid: "b59818fcaf734e5b85ce691bda0e822a"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "40e69efd62872e63413bb25e7f8006d6"
server_key_iv: "94b4b3a23e684cd495146cfd7e5154b8"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.812451 24547 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:28.815311 24547 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 15:54:28.815387 24547 log.cc:1196] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 15:54:28.815745 24547 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Deleting consensus metadata
I20260806 15:54:28.815943 24547 master_runner.cc:315] Copying system tablet from 127.13.223.62:44803
I20260806 15:54:28.817484 24547 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: tablet copy: Beginning tablet copy session from remote peer at address 127.13.223.62:44803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.60.keytab.
Entry for principal oryx/127.13.223.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.60.keytab.
I20260806 15:54:28.821682 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:39727
--webserver_interface=127.13.223.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.60.keytab
--principal=oryx/127.13.223.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:54:28.827593 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.818556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.13.223.62:44803)
0806 15:54:28.818712 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:28.818718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:28.818736 (+ 18us) server_negotiation.cc:407] Connection header received
0806 15:54:28.818788 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:28.818791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:28.818842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:28.818942 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:28.819450 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.820276 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.822135 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.822482 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.822592 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.824113 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:28.824116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:28.824121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:28.824158 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:28.825758 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.826225 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.826228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.826230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.826311 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:28.826592 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:28.826596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:28.826598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:28.826858 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:28.826966 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:28.827310 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 15:54:28.827426 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:28.827605 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:28.818156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52420)
0806 15:54:28.818412 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:28.818429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:28.818591 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:28.819052 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:28.819054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:28.819219 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:28.819326 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.819331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.820428 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.820431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.821972 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:28.821987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:28.822659 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:28.822663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:28.822821 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:28.823441 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:28.823460 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:28.823965 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:28.825893 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.825899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.825901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.826116 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.826402 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:28.826405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:28.826407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:28.826500 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:28.827032 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:28.827035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:28.827137 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 15:54:28.827251 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 15:54:28.827403 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260806 15:54:28.828703 23855 tablet_copy_service.cc:140] P 38e98a7e487340bab964f5dfbbd35b70: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b59818fcaf734e5b85ce691bda0e822a ({username='oryx', principal='oryx/127.13.223.61@KRBTEST.COM'} at 127.0.0.1:52420)
I20260806 15:54:28.828892 23855 tablet_copy_service.cc:161] P 38e98a7e487340bab964f5dfbbd35b70: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b59818fcaf734e5b85ce691bda0e822a at {username='oryx', principal='oryx/127.13.223.61@KRBTEST.COM'} at 127.0.0.1:52420: session id = b59818fcaf734e5b85ce691bda0e822a-00000000000000000000000000000000
I20260806 15:54:28.830394 23855 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 15:54:28.832192 24547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 15:54:28.834249 24547 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: tablet copy: Starting download of 0 data blocks...
I20260806 15:54:28.834507 24547 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: tablet copy: Starting download of 1 WAL segments...
I20260806 15:54:28.836525 24547 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 15:54:28.838299 23855 tablet_copy_service.cc:342] P 38e98a7e487340bab964f5dfbbd35b70: Request end of tablet copy session b59818fcaf734e5b85ce691bda0e822a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.223.61@KRBTEST.COM'} at 127.0.0.1:52420
I20260806 15:54:28.838424 23855 tablet_copy_service.cc:433] P 38e98a7e487340bab964f5dfbbd35b70: ending tablet copy session b59818fcaf734e5b85ce691bda0e822a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b59818fcaf734e5b85ce691bda0e822a
I20260806 15:54:28.845558 24547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:28.847271 24725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:28.847923 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:28.849905 24547 server_base.cc:1034] running on GCE node
W20260806 15:54:28.852691 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:28.852948 24547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:28.853294 24547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:28.854436 24547 hybrid_clock.cc:648] HybridClock initialized: now 1786031668854445 us; error 42 us; skew 500 ppm
Aug 06 15:54:28 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031668, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:28.855923 24547 init.cc:377] Logged in from keytab as oryx/127.13.223.61@KRBTEST.COM (short username oryx)
I20260806 15:54:28.856393 24547 webserver.cc:460] Webserver started at http://127.13.223.61:35175/ using document root <none> and password file <none>
I20260806 15:54:28.856613 24547 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:28.856673 24547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:28.857753 24547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:28.858703 24742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:28.858930 24547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:28.859020 24547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
uuid: "b59818fcaf734e5b85ce691bda0e822a"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "40e69efd62872e63413bb25e7f8006d6"
server_key_iv: "94b4b3a23e684cd495146cfd7e5154b8"
server_key_version: "encryptionkey@0"
I20260806 15:54:28.859120 24547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:28.870504 24547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:28.872460 24547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:28.879971 24547 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.61:39527
I20260806 15:54:28.880646 24815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.61:39527 every 8 connection(s)
I20260806 15:54:28.880702 24547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
I20260806 15:54:28.882315 24816 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:54:28.883113 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Bootstrap starting.
I20260806 15:54:28.892011 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:54:28.892310 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Bootstrap complete.
I20260806 15:54:28.892879 24816 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } }
I20260806 15:54:28.893020 24816 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b59818fcaf734e5b85ce691bda0e822a, State: Initialized, Role: LEARNER
I20260806 15:54:28.893142 24816 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } }
I20260806 15:54:28.893515 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } }
I20260806 15:54:28.893656 24816 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:28.893718 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: LEARNER
I20260806 15:54:28.896903 24837 catalog_manager.cc:1259] Loaded cluster ID: e96f23a66721400d9de599be83617ee0
I20260806 15:54:28.896978 24837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: loading cluster ID for follower catalog manager: success
I20260806 15:54:28.898600 24837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: acquiring CA information for follower catalog manager: success
I20260806 15:54:28.899690 24837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 15:54:29.018028 24724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:29.018348 24724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:29.018464 24724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:29.022804 24724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:29.022891 24724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:29.022933 24724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:29.022997 24724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:29.023041 24724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:29.028880 24724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.60.keytab
--principal=oryx/127.13.223.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:29.030093 24724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:29.031227 24724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:54:29.032934 24107 heartbeater.cc:503] Master 127.13.223.62:44803 was elected leader, sending a full tablet report...
W20260806 15:54:29.041218 24850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.041177 24844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.041179 24845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.041620 24724 server_base.cc:1034] running on GCE node
I20260806 15:54:29.042271 24724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:29.043324 24724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:29.044625 24724 hybrid_clock.cc:648] HybridClock initialized: now 1786031669044590 us; error 44 us; skew 500 ppm
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:29.047443 24724 init.cc:377] Logged in from keytab as oryx/127.13.223.60@KRBTEST.COM (short username oryx)
I20260806 15:54:29.048261 24724 webserver.cc:460] Webserver started at http://127.13.223.60:33603/ using document root <none> and password file <none>
I20260806 15:54:29.048854 24724 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:29.048930 24724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:29.049098 24724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:29.051241 24724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/instance:
uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
format_stamp: "Formatted at 2026-08-06 15:54:29 on dist-test-slave-rwrg"
server_key: "c62da085086bb9289a6076e873fdc8d7"
server_key_iv: "00b6bb7beadddc91df4885779ece8371"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.051806 24724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal/instance:
uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
format_stamp: "Formatted at 2026-08-06 15:54:29 on dist-test-slave-rwrg"
server_key: "c62da085086bb9289a6076e873fdc8d7"
server_key_iv: "00b6bb7beadddc91df4885779ece8371"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.055939 24724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:29.058398 24862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.059846 24724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:29.059983 24724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
format_stamp: "Formatted at 2026-08-06 15:54:29 on dist-test-slave-rwrg"
server_key: "c62da085086bb9289a6076e873fdc8d7"
server_key_iv: "00b6bb7beadddc91df4885779ece8371"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.060093 24724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:29.074362 24724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:29.077315 24724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:29.077571 24724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:29.092375 24724 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.60:39727
I20260806 15:54:29.092746 24947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.60:39727 every 8 connection(s)
I20260806 15:54:29.093780 24724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
I20260806 15:54:29.097138 24948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:29.101819 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 24724
I20260806 15:54:29.101786 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.60@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.60@KRBTEST.COM
I20260806 15:54:29.118376 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.105126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.13.223.62:44803)
0806 15:54:29.105328 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.105335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.105354 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:54:29.105886 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.105893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.105974 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.106087 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.107449 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.108272 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.110215 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.110586 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.110705 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.113667 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.113674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.113680 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.113719 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.115756 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.116450 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.116457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.116461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.116614 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.117055 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.117059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.117060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.117370 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.117504 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.118063 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.118189 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 15:54:29.119253 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.103800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52436)
0806 15:54:29.104379 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.104429 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.105603 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.106338 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.106347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.106864 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.107276 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.107295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.108471 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.108478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.110046 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.110059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.110776 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.110782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.110979 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.111602 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.111619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.113495 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.115905 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.115917 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.115932 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.116302 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.116739 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.116743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.116745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.116924 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.117487 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.117493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.117810 (+ 317us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.117998 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.118335 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":6,"thread_start_us":183,"threads_started":1}
I20260806 15:54:29.120568 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.103440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.13.223.60:39727)
0806 15:54:29.104004 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.104042 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.104108 (+ 66us) server_negotiation.cc:407] Connection header received
0806 15:54:29.105593 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.105619 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.105889 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.106082 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.106937 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.108667 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.109526 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.110215 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.110375 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.114363 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.114369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.114380 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.114401 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.116739 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.117248 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.117253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.117254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.117322 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.117648 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.117651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.117653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.118071 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.119999 (+ 1928us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.120112 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.120267 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":414,"thread_start_us":157,"threads_started":1}
I20260806 15:54:29.120754 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.60@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
I20260806 15:54:29.130539 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.121299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45320)
0806 15:54:29.121958 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.121975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.122144 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.122517 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.122523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.122788 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.122932 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.122942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.123928 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.123932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.124466 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.124474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.125049 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.125052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.125227 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.125747 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.125758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.127184 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.129065 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.129070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.129072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.129279 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.129565 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.129569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.129571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.129655 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.130032 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.130035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.130132 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.130239 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.130367 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563}
I20260806 15:54:29.130574 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.121628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.13.223.61:39527)
0806 15:54:29.121996 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.122001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.122018 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:54:29.122313 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.122317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.122375 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.122479 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.123092 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.123777 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.124702 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.124933 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.125053 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.127317 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.127321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.127323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.127349 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.128936 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.129382 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.129386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.129388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.129471 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.129748 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.129752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.129754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.129935 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.130043 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.130294 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.130412 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":161,"threads_started":1}
I20260806 15:54:29.131393 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.60@KRBTEST.COM for oryx/127.13.223.60@KRBTEST.COM
I20260806 15:54:29.138731 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.131838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.0.0.1:46174)
0806 15:54:29.131945 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.131959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.132050 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.132321 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.132322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.132438 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.132543 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.132549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.133251 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.133253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.133728 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.133734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.134152 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.134154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.134207 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.134653 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.134664 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.135774 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.137288 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.137295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.137297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.137478 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.137753 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.137759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.137761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.137868 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.138222 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.138225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.138304 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.138417 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.138546 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 15:54:29.138731 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.132044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.13.223.60:39727)
0806 15:54:29.132155 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.132158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.132169 (+ 11us) server_negotiation.cc:407] Connection header received
0806 15:54:29.132196 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.132198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.132229 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.132292 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.132658 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.133158 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.133843 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.134058 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.134138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.135894 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.135897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.135900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.135924 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.137181 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.137573 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.137578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.137580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.137652 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.137941 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.137946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.137948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.138128 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.138233 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.138461 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.138603 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":49}
I20260806 15:54:29.142161 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Bootstrap starting.
I20260806 15:54:29.144796 24948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:29.145740 24948 log.cc:824] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:29.148173 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: No bootstrap required, opened a new log
I20260806 15:54:29.151716 24948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } }
I20260806 15:54:29.151911 24948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:29.151989 24948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad34f7a3f0b4b3f9ca3d16660b96d3b, State: Initialized, Role: FOLLOWER
I20260806 15:54:29.152448 24948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } }
I20260806 15:54:29.153190 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } } }
I20260806 15:54:29.153365 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:29.153733 24948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: configured and running, proceeding with master startup.
W20260806 15:54:29.159097 24978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:54:29.159193 24978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:54:29.162807 24724 master_runner.cc:428] Detected that this master 2ad34f7a3f0b4b3f9ca3d16660b96d3b is joining an existing cluster
I20260806 15:54:29.162864 24724 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.223.60:39727
I20260806 15:54:29.174108 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 23918
I20260806 15:54:29.190738 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.165521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.13.223.61:39527)
0806 15:54:29.165687 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.165694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.166734 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 15:54:29.166918 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.166921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.166980 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.167033 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.167912 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.168805 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.171087 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.171450 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.171527 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.186804 (+ 15277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.186807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.186810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.186838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.189073 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.189546 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.189549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.189550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.189624 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.189876 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.189879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.189880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.190075 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.190135 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.190427 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.190565 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 15:54:29.191574 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.165730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45322)
0806 15:54:29.166622 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.166636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.166804 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.167190 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.167193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.167581 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.167762 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.167771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.168952 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.168955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.170938 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.170947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.171691 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.171694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.171808 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.172251 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.172268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.186318 (+ 14050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.189209 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.189212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.189214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.189438 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.189717 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.189720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.189721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.189789 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.190198 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.190200 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.190285 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.191156 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.191299 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":69,"thread_start_us":127,"threads_started":1}
I20260806 15:54:29.203702 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.166092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.13.223.60:39727)
0806 15:54:29.166218 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.166223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.166724 (+ 501us) server_negotiation.cc:407] Connection header received
0806 15:54:29.166861 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.166864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.166897 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.166939 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.167805 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.168625 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.170121 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.170482 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.170601 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.171910 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.171913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.171916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.171941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.186121 (+ 14180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.202240 (+ 16119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.202244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.202246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.202339 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.202640 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.202643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.202645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.202863 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.202955 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.203391 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.203532 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":47}
I20260806 15:54:29.204525 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.166170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.0.0.1:46178)
0806 15:54:29.166654 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.166667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.166772 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.167161 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.167167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.167448 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.167682 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.167692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.168777 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.168779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.169996 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.170006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.170639 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.170643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.170836 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.171389 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.171411 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.171766 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.201709 (+ 29943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.201713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.201715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.202026 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.202447 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.202450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.202452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.202548 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.203012 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.203015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.203095 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.204129 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.204294 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":184,"thread_start_us":190,"threads_started":1}
I20260806 15:54:29.204860 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.165410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.13.223.62:44803)
0806 15:54:29.165523 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.165533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.166253 (+ 720us) server_negotiation.cc:407] Connection header received
0806 15:54:29.166530 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.166534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.166599 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.166655 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.167823 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.168625 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.170141 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.170474 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.170575 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.187154 (+ 16579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.187158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.187160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.187190 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.202476 (+ 15286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.203147 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.203154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.203158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.203261 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.203610 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.203616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.203621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.203874 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.203956 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.204368 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.204610 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 15:54:29.205595 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.165418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52448)
0806 15:54:29.166106 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.166123 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.166369 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.166851 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.166853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.167390 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.167682 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.167692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.168767 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.168770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.169995 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.170006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.170741 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.170744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.170925 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.185475 (+ 14550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.185519 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.186644 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.202697 (+ 16053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.202703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.202708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.203020 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.203383 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.203389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.203394 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.203488 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.204046 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.204052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.204163 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.205085 (+ 922us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.205278 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":120,"thread_start_us":141,"threads_started":1}
I20260806 15:54:29.208956 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:35767
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:44803
--webserver_port=41263
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031667, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.62@KRBTEST.COM for oryx/127.13.223.60@KRBTEST.COM
I20260806 15:54:29.219026 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.209893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.13.223.60:39727)
0806 15:54:29.210022 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.210028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.210149 (+ 121us) server_negotiation.cc:407] Connection header received
0806 15:54:29.210301 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.210305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.210360 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.210436 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.210988 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.211487 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.212366 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.212684 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.212795 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.215363 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.215367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.215370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.215396 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.217123 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.217668 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.217672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.217674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.217760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.218094 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.218098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.218099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.218303 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.218401 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.218754 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.218869 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 15:54:29.219092 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.209798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.0.0.1:46192)
0806 15:54:29.210061 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.210079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.210176 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.210480 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.210486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.210705 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.210833 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.210843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.211615 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.211620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.212246 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.212256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.212799 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.212803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.212918 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.213433 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.213457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.215203 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.217249 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.217255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.217260 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.217555 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.217868 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.217875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.217879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.217989 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.218416 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.218422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.218558 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.218688 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.218861 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 15:54:29.220440 23823 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.223.60:39727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
member_type: NON_VOTER
last_known_addr {
host: "127.13.223.60"
port: 39727
}
attrs {
promote: true
}
}
dest_uuid: "38e98a7e487340bab964f5dfbbd35b70"
cas_config_opid_index: 5
I20260806 15:54:29.221318 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } }
I20260806 15:54:29.222656 24701 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.224212 24701 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.223.60:39727
I20260806 15:54:29.224376 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.224474 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.225271 24724 master.cc:537] Master@127.13.223.60:39727 shutting down...
W20260806 15:54:29.226325 23803 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 -> Peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60:39727): Couldn't send request to peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 15:54:29.231226 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.224896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45334)
0806 15:54:29.226724 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.226738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.226822 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.227127 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.227130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.227149 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.227239 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.227246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.228354 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.228358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.229871 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.229883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.230805 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.230809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.230896 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.230943 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.231023 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.231094 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755}
I20260806 15:54:29.231235 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.224989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.13.223.61:39527)
0806 15:54:29.225107 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.225112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.226817 (+ 1705us) server_negotiation.cc:407] Connection header received
0806 15:54:29.226910 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.226915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.226989 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.227087 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.227368 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.228225 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.230010 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.230678 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.230789 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.230939 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.231034 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.231101 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 15:54:29.231712 24778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 LEARNER]: Refusing update from remote peer 38e98a7e487340bab964f5dfbbd35b70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 15:54:29.232148 24703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 15:54:29.234091 24778 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.236128 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e98a7e487340bab964f5dfbbd35b70. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.236254 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: LEARNER
I20260806 15:54:29.236328 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: NON_VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: true } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.236387 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: LEARNER
I20260806 15:54:29.236583 24703 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: attempting to promote NON_VOTER b59818fcaf734e5b85ce691bda0e822a to VOTER
I20260806 15:54:29.237025 24703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } }
I20260806 15:54:29.238826 24778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 LEARNER]: Refusing update from remote peer 38e98a7e487340bab964f5dfbbd35b70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 15:54:29.238977 24724 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:29.239254 24724 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:29.239369 24724 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: stopping tablet replica
I20260806 15:54:29.239631 24701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 15:54:29.240010 23803 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 -> Peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60:39727): Couldn't send request to peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 15:54:29.241899 24703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b59818fcaf734e5b85ce691bda0e822a (127.13.223.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.243332 24703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.243460 24703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.243829 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.243925 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.244159 24778 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b59818fcaf734e5b85ce691bda0e822a (127.13.223.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.245714 24994 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.245898 24994 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:29.249747 24335 heartbeater.cc:503] Master 127.13.223.62:44803 was elected leader, sending a full tablet report...
I20260806 15:54:29.257200 24724 master.cc:552] Master@127.13.223.60:39727 shutdown complete.
I20260806 15:54:29.257251 24724 master_runner.cc:305] Clearing existing system tablet
I20260806 15:54:29.257485 24724 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:29.257542 24724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:29.258831 24724 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:29.259740 25001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.259964 24724 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:29.260210 24724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:29.260289 24724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
format_stamp: "Formatted at 2026-08-06 15:54:29 on dist-test-slave-rwrg"
server_key: "c62da085086bb9289a6076e873fdc8d7"
server_key_iv: "00b6bb7beadddc91df4885779ece8371"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.262382 24724 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:29.264973 24724 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 15:54:29.265127 24724 log.cc:1196] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 15:54:29.265496 24724 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Deleting consensus metadata
I20260806 15:54:29.265724 24724 master_runner.cc:315] Copying system tablet from 127.13.223.62:44803
I20260806 15:54:29.266989 24724 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: tablet copy: Beginning tablet copy session from remote peer at address 127.13.223.62:44803
I20260806 15:54:29.276348 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.267538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52458)
0806 15:54:29.267782 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.267798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.267906 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.268243 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.268246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.268535 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.268697 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.268705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.269810 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.269813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.270797 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.270812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.271435 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.271440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.271585 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.272211 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.272226 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.272708 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.274731 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.274734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.274737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.274987 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.275287 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.275290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.275292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.275385 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.275821 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.275824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.275920 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.276024 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.276152 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260806 15:54:29.276360 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.267901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.13.223.62:44803)
0806 15:54:29.268010 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.268015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.268029 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:29.268071 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.268074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.268120 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.268205 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.268825 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.269681 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.270937 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.271293 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.271401 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.272880 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.272884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.272887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.272915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.274613 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.275093 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.275097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.275099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.275186 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.275485 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.275489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.275490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.275720 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.275813 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.276096 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.276210 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:54:29.277144 23855 tablet_copy_service.cc:140] P 38e98a7e487340bab964f5dfbbd35b70: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b ({username='oryx', principal='oryx/127.13.223.60@KRBTEST.COM'} at 127.0.0.1:52458)
I20260806 15:54:29.277232 23855 tablet_copy_service.cc:161] P 38e98a7e487340bab964f5dfbbd35b70: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b at {username='oryx', principal='oryx/127.13.223.60@KRBTEST.COM'} at 127.0.0.1:52458: session id = 2ad34f7a3f0b4b3f9ca3d16660b96d3b-00000000000000000000000000000000
I20260806 15:54:29.278291 23855 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 15:54:29.279458 24724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 15:54:29.281828 24724 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: tablet copy: Starting download of 0 data blocks...
I20260806 15:54:29.282110 24724 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: tablet copy: Starting download of 1 WAL segments...
I20260806 15:54:29.283684 24724 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 15:54:29.285395 23855 tablet_copy_service.cc:342] P 38e98a7e487340bab964f5dfbbd35b70: Request end of tablet copy session 2ad34f7a3f0b4b3f9ca3d16660b96d3b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.223.60@KRBTEST.COM'} at 127.0.0.1:52458
I20260806 15:54:29.285476 23855 tablet_copy_service.cc:433] P 38e98a7e487340bab964f5dfbbd35b70: ending tablet copy session 2ad34f7a3f0b4b3f9ca3d16660b96d3b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2ad34f7a3f0b4b3f9ca3d16660b96d3b
I20260806 15:54:29.293210 24724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:29.295035 25013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.295141 25020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.295697 25015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.298038 24724 server_base.cc:1034] running on GCE node
I20260806 15:54:29.299183 24724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:29.299486 24724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:29.302034 24724 hybrid_clock.cc:648] HybridClock initialized: now 1786031669302022 us; error 29 us; skew 500 ppm
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:29.303730 24724 init.cc:377] Logged in from keytab as oryx/127.13.223.60@KRBTEST.COM (short username oryx)
I20260806 15:54:29.304224 24724 webserver.cc:460] Webserver started at http://127.13.223.60:45905/ using document root <none> and password file <none>
I20260806 15:54:29.304411 24724 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:29.304454 24724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:29.305878 24724 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:29.306970 25032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.307237 24724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:29.307313 24724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
format_stamp: "Formatted at 2026-08-06 15:54:29 on dist-test-slave-rwrg"
server_key: "c62da085086bb9289a6076e873fdc8d7"
server_key_iv: "00b6bb7beadddc91df4885779ece8371"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.307391 24724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:29.318997 24724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:29.321230 24724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:29.326939 24724 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.60:39727
I20260806 15:54:29.326968 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.60:39727 every 8 connection(s)
I20260806 15:54:29.327556 24724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
I20260806 15:54:29.330535 25114 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:54:29.331611 25114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Bootstrap starting.
I20260806 15:54:29.343603 25114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:54:29.343932 25114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: Bootstrap complete.
I20260806 15:54:29.344538 25114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } }
I20260806 15:54:29.344692 25114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ad34f7a3f0b4b3f9ca3d16660b96d3b, State: Initialized, Role: LEARNER
I20260806 15:54:29.344815 25114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } }
I20260806 15:54:29.345176 25118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.345311 25114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:29.345397 25118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: This master's current role is: LEARNER
I20260806 15:54:29.349090 25141 catalog_manager.cc:1259] Loaded cluster ID: e96f23a66721400d9de599be83617ee0
I20260806 15:54:29.349218 25141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: loading cluster ID for follower catalog manager: success
I20260806 15:54:29.351318 25141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: acquiring CA information for follower catalog manager: success
I20260806 15:54:29.352685 25141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 15:54:29.359624 24991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:29.359910 24991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:29.360005 24991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:29.364198 24991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:29.364254 24991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:29.364342 24991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:29.370141 24991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:35767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=41263
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:29.371325 24991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:29.372429 24991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:29.382134 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.382238 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.382445 24991 server_base.cc:1034] running on GCE node
W20260806 15:54:29.382238 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.382902 24991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:29.383530 24991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:29.384716 24991 hybrid_clock.cc:648] HybridClock initialized: now 1786031669384698 us; error 29 us; skew 500 ppm
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:29.387611 24991 init.cc:377] Logged in from keytab as oryx/127.13.223.1@KRBTEST.COM (short username oryx)
I20260806 15:54:29.388578 24991 webserver.cc:460] Webserver started at http://127.13.223.1:41263/ using document root <none> and password file <none>
I20260806 15:54:29.389178 24991 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:29.389312 24991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:29.393606 24991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:54:29.395804 25166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.397111 24991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:29.397248 24991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "95c9f851accc49cd900e4b1c97970ace"
format_stamp: "Formatted at 2026-08-06 15:54:27 on dist-test-slave-rwrg"
server_key: "f54dce5bfab111ed5e1bb9535f5fe6a9"
server_key_iv: "f631d87dc7f469679791b8d1e9dc6e17"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.397822 24991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:29.413499 24991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:29.416986 24991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:29.417253 24991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:29.417922 24991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:29.418819 24991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:29.418895 24991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.418972 24991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:29.419016 24991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.432797 24991 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:35767
I20260806 15:54:29.432844 25346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:35767 every 8 connection(s)
I20260806 15:54:29.433856 24991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:29.442231 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 24991
I20260806 15:54:29.442410 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 24117
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.60@KRBTEST.COM
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
I20260806 15:54:29.454523 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.438071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:37527 (local address 127.13.223.60:39727)
0806 15:54:29.438343 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.438350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.438368 (+ 18us) server_negotiation.cc:407] Connection header received
0806 15:54:29.439177 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.439180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.439232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.439346 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.440451 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.441503 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.443268 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.443795 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.444488 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.448782 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.448788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.448794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.448826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.451642 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.452224 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.452228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.452229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.452344 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.452865 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.452869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.452871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.453118 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.453247 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.454162 (+ 915us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.454318 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":121,"threads_started":1}
I20260806 15:54:29.454782 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.435914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:36155 (local address 127.13.223.61:39527)
0806 15:54:29.436031 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.436035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.437446 (+ 1411us) server_negotiation.cc:407] Connection header received
0806 15:54:29.438916 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.438919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.438975 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.439051 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.440454 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.441282 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.443236 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.443547 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.443621 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.449537 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.449543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.449548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.449580 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.452244 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.452781 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.452785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.452787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.452878 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.453248 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.453252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.453255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.453505 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.453585 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.454249 (+ 664us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.454418 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:54:29.455442 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.437642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.13.223.1:37527)
0806 15:54:29.437979 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.437995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.438567 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.439385 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.439388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.439769 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.440292 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.440308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.441616 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.441620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.443102 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.443117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.444014 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.444022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.444201 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.445716 (+ 1515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.445747 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.448571 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.451794 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.451800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.451818 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.452092 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.452439 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.452445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.452450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.452738 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.453631 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.453637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.453919 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.454120 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.454468 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":387,"spinlock_wait_cycles":137728,"thread_start_us":110,"threads_started":1}
I20260806 15:54:29.455893 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.436455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.13.223.1:36155)
0806 15:54:29.437142 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.437205 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.438553 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.439118 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.439125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.439769 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.440292 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.440308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.441390 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.441393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.443112 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.443121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.443795 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.443798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.443905 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.445474 (+ 1569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.445510 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.449385 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.452372 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.452378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.452383 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.452661 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.452975 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.452981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.452986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.453108 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.453630 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.453641 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.453936 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.454965 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.455161 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":557,"spinlock_wait_cycles":67456,"thread_start_us":195,"threads_started":1}
I20260806 15:54:29.456831 25348 heartbeater.cc:347] Connected to a master server at 127.13.223.60:39727
I20260806 15:54:29.457047 25348 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.457494 25348 heartbeater.cc:511] Master 127.13.223.60:39727 requested a full tablet report, sending...
I20260806 15:54:29.457973 25352 heartbeater.cc:347] Connected to a master server at 127.13.223.61:39527
I20260806 15:54:29.458045 25352 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.458206 25352 heartbeater.cc:511] Master 127.13.223.61:39527 requested a full tablet report, sending...
I20260806 15:54:29.459146 25055 ts_manager.cc:194] Registered new tserver with Master: 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767)
I20260806 15:54:29.459256 24765 ts_manager.cc:194] Registered new tserver with Master: 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767)
I20260806 15:54:29.466419 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.435832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:37325 (local address 127.13.223.62:44803)
0806 15:54:29.435982 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.435987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.437446 (+ 1459us) server_negotiation.cc:407] Connection header received
0806 15:54:29.438784 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.438788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.438851 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.438933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.440440 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.441045 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.442956 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.443308 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.443378 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.449516 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.449523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.449529 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.449583 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.464471 (+ 14888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.465082 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.465085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.465087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.465184 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.465528 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.465532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.465534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.465736 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.465864 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.466137 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.466261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:54:29.466464 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.436455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:37325)
0806 15:54:29.437142 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.437212 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.438556 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.439121 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.439128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.439742 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.440288 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.440302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.441237 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.441240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.442812 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.442824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.443569 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.443575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.443798 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.445554 (+ 1756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.445910 (+ 356us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.449302 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.464715 (+ 15413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.464719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.464721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.464948 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.465286 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.465290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.465292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.465384 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.465853 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.465856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.465961 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.466072 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.466232 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":120,"spinlock_wait_cycles":823424,"thread_start_us":174,"threads_started":1}
I20260806 15:54:29.467070 25350 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44803
I20260806 15:54:29.467187 25350 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.467368 25350 heartbeater.cc:511] Master 127.13.223.62:44803 requested a full tablet report, sending...
I20260806 15:54:29.468202 23823 ts_manager.cc:194] Re-registered known tserver with Master: 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767)
I20260806 15:54:29.468910 23823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:37325
I20260806 15:54:29.469877 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:34903
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:44803
--webserver_port=42311
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:54:29.492592 24539 heartbeater.cc:503] Master 127.13.223.62:44803 was elected leader, sending a full tablet report...
W20260806 15:54:29.585647 25363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:29.585920 25363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:29.585991 25363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:29.590085 25363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:29.590178 25363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:29.590269 25363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:29.596421 25363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:34903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=42311
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:29.597788 25363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:29.598919 25363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:29.609416 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.609402 25369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.609398 25372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.609764 25363 server_base.cc:1034] running on GCE node
I20260806 15:54:29.610179 25363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:29.610841 25363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:29.612007 25363 hybrid_clock.cc:648] HybridClock initialized: now 1786031669611989 us; error 29 us; skew 500 ppm
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:29.614866 25363 init.cc:377] Logged in from keytab as oryx/127.13.223.2@KRBTEST.COM (short username oryx)
I20260806 15:54:29.615716 25363 webserver.cc:460] Webserver started at http://127.13.223.2:42311/ using document root <none> and password file <none>
I20260806 15:54:29.616287 25363 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:29.616387 25363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:29.620309 25363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:29.622588 25387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.623854 25363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:29.623991 25363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "b418ae20483d4f17aab8b7620bd13f02"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "f81bc32d270a8327bd2a22194ea8e1cb"
server_key_iv: "abf439ab7ee5a019d2d70cc811df0d00"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.624675 25363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:29.637027 25363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:29.640339 25363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:29.640625 25363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:29.641235 25363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:29.642079 25363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:29.642151 25363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.642230 25363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:29.642272 25363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.652920 25363 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:34903
I20260806 15:54:29.652941 25558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:34903 every 8 connection(s)
I20260806 15:54:29.653924 25363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:29.657624 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 25363
I20260806 15:54:29.657790 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 24342
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
I20260806 15:54:29.675151 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.655779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:57859 (local address 127.13.223.62:44803)
0806 15:54:29.655946 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.655950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.656855 (+ 905us) server_negotiation.cc:407] Connection header received
0806 15:54:29.661613 (+ 4758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.661620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.661696 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.661767 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.663721 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.664538 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.666304 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.666641 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.666715 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.670502 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.670506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.670509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.670541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.672439 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.673277 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.673281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.673283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.673380 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.673797 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.673800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.673802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.674013 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.674101 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.674841 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.674970 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 15:54:29.677014 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.656174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:57859)
0806 15:54:29.656694 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.656746 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.661299 (+ 4553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.662291 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.662302 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.662817 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.663563 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.663581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.664686 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.664691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.666124 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.666137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.666965 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.666971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.667222 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.667982 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.668006 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.670287 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.672600 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.672608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.672629 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.673149 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.673492 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.673495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.673498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.673684 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.674178 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.674188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.674550 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.675342 (+ 792us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.675720 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"spinlock_wait_cycles":2304,"thread_start_us":110,"threads_started":1}
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.60@KRBTEST.COM
I20260806 15:54:29.678622 25560 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44803
I20260806 15:54:29.678903 25560 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.679083 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.655705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:57517 (local address 127.13.223.61:39527)
0806 15:54:29.655833 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.655838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.656861 (+ 1023us) server_negotiation.cc:407] Connection header received
0806 15:54:29.662533 (+ 5672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.662537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.662593 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.662670 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.668873 (+ 6203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.669838 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.671138 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.671523 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.671649 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.674728 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.674735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.674741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.674779 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.676710 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.677269 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.677275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.677280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.677369 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.677673 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.677679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.677683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.677936 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.678028 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.678755 (+ 727us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.678896 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:54:29.679515 25560 heartbeater.cc:511] Master 127.13.223.62:44803 requested a full tablet report, sending...
I20260806 15:54:29.679527 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.656173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.13.223.2:57517)
0806 15:54:29.656693 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.656747 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.662187 (+ 5440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.668023 (+ 5836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.668026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.668420 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.668633 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.668641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.670108 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.670111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.670986 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.671000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.671827 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.671832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.671946 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.672548 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.672569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.674540 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.676837 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.676842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.676847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.677152 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.677468 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.677474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.677479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.677569 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.678075 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.678081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.678200 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.678297 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.678463 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":45,"spinlock_wait_cycles":18304,"thread_start_us":137,"threads_started":1}
I20260806 15:54:29.679914 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:40601
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:44803
--webserver_port=32957
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:54:29.680487 25562 heartbeater.cc:347] Connected to a master server at 127.13.223.61:39527
I20260806 15:54:29.680672 25562 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.680881 25562 heartbeater.cc:511] Master 127.13.223.61:39527 requested a full tablet report, sending...
I20260806 15:54:29.681357 23823 ts_manager.cc:194] Re-registered known tserver with Master: b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903)
I20260806 15:54:29.681658 24765 ts_manager.cc:194] Registered new tserver with Master: b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903)
I20260806 15:54:29.682129 23823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:57859
I20260806 15:54:29.683889 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.657171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.13.223.2:42243)
0806 15:54:29.657437 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.657451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.672186 (+ 14735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.672681 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.672687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.672928 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.673063 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.673072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.674236 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.674239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.675069 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.675079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.676117 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.676121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.676213 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.676698 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.676723 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.679771 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.682212 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.682218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.682222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.682516 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.682858 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.682863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.682865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.682956 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.683349 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.683352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.683450 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.683553 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.683720 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260806 15:54:29.683892 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.657396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:42243 (local address 127.13.223.60:39727)
0806 15:54:29.657532 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.657537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.657553 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:29.672366 (+ 14813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.672371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.672441 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.672522 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.673232 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.674025 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.675208 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.675527 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.675604 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.679930 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.679934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.679936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.679966 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.681765 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.682648 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.682653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.682655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.682755 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.683054 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.683058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.683059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.683250 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.683360 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.683619 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.683752 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 15:54:29.684718 25559 heartbeater.cc:347] Connected to a master server at 127.13.223.60:39727
I20260806 15:54:29.684830 25559 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.685010 25559 heartbeater.cc:511] Master 127.13.223.60:39727 requested a full tablet report, sending...
I20260806 15:54:29.685737 25055 ts_manager.cc:194] Registered new tserver with Master: b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903)
W20260806 15:54:29.801045 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:29.801311 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:29.801395 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:29.805555 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:29.805629 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:29.805688 25568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:29.811206 25568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:40601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=32957
--tserver_master_addrs=127.13.223.62:44803,127.13.223.61:39527,127.13.223.60:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:29.812237 25568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:29.813295 25568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:29.823303 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:29.823305 25579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.823393 25568 server_base.cc:1034] running on GCE node
W20260806 15:54:29.823350 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:29.824175 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:29.824951 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:29.826185 25568 hybrid_clock.cc:648] HybridClock initialized: now 1786031669826167 us; error 38 us; skew 500 ppm
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:29.829888 25568 init.cc:377] Logged in from keytab as oryx/127.13.223.3@KRBTEST.COM (short username oryx)
I20260806 15:54:29.830919 25568 webserver.cc:460] Webserver started at http://127.13.223.3:32957/ using document root <none> and password file <none>
I20260806 15:54:29.831597 25568 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:29.831737 25568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:29.836282 25568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:29.839208 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.840821 25568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:29.840984 25568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "a74caae195ad48cabe9946b38b7afe6b"
format_stamp: "Formatted at 2026-08-06 15:54:28 on dist-test-slave-rwrg"
server_key: "0f17e6aa167577c6078155b1c68074b0"
server_key_iv: "b19d7ba3c334142622dd6469b91eda67"
server_key_version: "encryptionkey@0"
I20260806 15:54:29.841682 25568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:29.862211 25568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:29.863193 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.858470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.0.0.1:46208)
0806 15:54:29.858827 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.858842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.858944 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.859589 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.859595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.859623 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.859738 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.859745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.860905 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.860907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.861688 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.861697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.862716 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.862720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.862829 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.862852 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.862914 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.863001 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
I20260806 15:54:29.863174 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.858614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.13.223.60:39727)
0806 15:54:29.859198 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.859205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.859228 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:54:29.859301 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.859304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.859381 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.859540 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.859883 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.860782 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.861840 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.862548 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.862687 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.862855 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.862948 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.863054 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 15:54:29.863992 25071 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 15:54:29.864569 25602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e98a7e487340bab964f5dfbbd35b70. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.864765 25602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: This master's current role is: LEARNER
I20260806 15:54:29.866605 25568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:29.866811 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:29.867509 25568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:29.868599 25568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:29.868659 25568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.868708 25568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:29.868736 25568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:29.881151 25770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40601 every 8 connection(s)
I20260806 15:54:29.881165 25568 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40601
I20260806 15:54:29.882450 25568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:29.888805 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 25568
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.61@KRBTEST.COM
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.60@KRBTEST.COM
Aug 06 15:54:29 dist-test-slave-rwrg krb5kdc[23752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031669, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:29.902041 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.885031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:43377 (local address 127.13.223.60:39727)
0806 15:54:29.885158 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.885162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.885774 (+ 612us) server_negotiation.cc:407] Connection header received
0806 15:54:29.887046 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.887050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.887115 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.887173 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.889128 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.889975 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892114 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.892462 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892595 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.897029 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.897033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.897038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.897068 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.899271 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.899932 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.899935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.899937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.900031 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.900517 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.900521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.900523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.900812 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.900909 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.901709 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.901851 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:54:29.903903 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.885077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.13.223.3:43377)
0806 15:54:29.885610 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.885664 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.886792 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.887500 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.887510 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.888123 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.888944 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.888965 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.890175 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.890179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.891972 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.891978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.892813 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.893168 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.894090 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.894114 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.896791 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.899415 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.899423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.899441 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.899799 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.900168 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.900173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.900178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.900371 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.901017 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.901025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.901374 (+ 349us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.902280 (+ 906us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.902718 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":514,"thread_start_us":129,"threads_started":1}
I20260806 15:54:29.905197 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.884593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:50077 (local address 127.13.223.61:39527)
0806 15:54:29.884768 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.884784 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.886178 (+ 1394us) server_negotiation.cc:407] Connection header received
0806 15:54:29.887158 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.887164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.887212 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.887285 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.889128 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.889976 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892119 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.892486 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892606 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.897731 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.897738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.897741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.897772 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.901941 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.903479 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.903483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.903484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.903573 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.903954 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.903959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.903964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.904191 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.904286 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.904901 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.905057 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 15:54:29.905288 25773 heartbeater.cc:347] Connected to a master server at 127.13.223.60:39727
I20260806 15:54:29.905545 25773 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.905884 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.885078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.13.223.3:50077)
0806 15:54:29.886080 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.886093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.886792 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.887500 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.887510 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.888123 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.888988 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.888995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.890185 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.890187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.891949 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.891964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.892813 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.893213 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.894428 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.894445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.897174 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.902853 (+ 5679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.902856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.902858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.903361 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.903747 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.903750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.903752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.903842 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.904336 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.904339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.904433 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.904546 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.905532 (+ 986us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":480,"spinlock_wait_cycles":98048,"thread_start_us":157,"threads_started":1}
I20260806 15:54:29.906205 25773 heartbeater.cc:511] Master 127.13.223.60:39727 requested a full tablet report, sending...
I20260806 15:54:29.907349 25777 heartbeater.cc:347] Connected to a master server at 127.13.223.61:39527
I20260806 15:54:29.907398 25055 ts_manager.cc:194] Registered new tserver with Master: a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601)
I20260806 15:54:29.907434 25777 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.907521 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.886445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:46091 (local address 127.13.223.62:44803)
0806 15:54:29.886582 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.886586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.886600 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:29.887053 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.887057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.887128 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.887183 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.889141 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.889956 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892138 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.892533 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892619 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.899376 (+ 6757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.899380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.899383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.899410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.901401 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.906010 (+ 4609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.906016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.906021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.906128 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.906479 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.906482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.906486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.906727 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.906817 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.907225 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.907349 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:54:29.907599 25777 heartbeater.cc:511] Master 127.13.223.61:39527 requested a full tablet report, sending...
I20260806 15:54:29.908119 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.886110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:46091)
0806 15:54:29.886378 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.886389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.886811 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.887500 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.887510 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.888083 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:29.888943 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.888967 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.890168 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.890172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.891977 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.891988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.892878 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.892882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.893182 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.894236 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:29.894251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:29.899202 (+ 4951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:29.905604 (+ 6402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.905608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.905611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.905883 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.906266 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:29.906271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:29.906276 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:29.906370 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:29.906882 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:29.906923 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:29.907019 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.907727 (+ 708us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.907880 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":386,"spinlock_wait_cycles":41088,"thread_start_us":85,"threads_started":1}
I20260806 15:54:29.908295 24765 ts_manager.cc:194] Registered new tserver with Master: a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601)
I20260806 15:54:29.908730 25774 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44803
I20260806 15:54:29.908806 25774 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:29.908816 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.897116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.13.223.62:44803)
0806 15:54:29.897388 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.897392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.898238 (+ 846us) server_negotiation.cc:407] Connection header received
0806 15:54:29.898457 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.898460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.898516 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.898676 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.899706 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.900571 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.901788 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.902159 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.902233 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.904691 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.904695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.904697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.904728 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.906760 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.907341 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.907344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.907346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.907424 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.907753 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.907756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.907758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.907975 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.908115 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.908543 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.908675 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260806 15:54:29.908931 25774 heartbeater.cc:511] Master 127.13.223.62:44803 requested a full tablet report, sending...
I20260806 15:54:29.909510 23823 ts_manager.cc:194] Re-registered known tserver with Master: a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601)
I20260806 15:54:29.910241 23823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:46091
I20260806 15:54:29.914788 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.897763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.13.223.60:39727)
0806 15:54:29.898275 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.898279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.900470 (+ 2191us) server_negotiation.cc:407] Connection header received
0806 15:54:29.900529 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.900532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.900582 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.900663 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.901560 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.902451 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.903640 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.904078 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.904171 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.910795 (+ 6624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.910798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.910801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.910825 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.912815 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.913287 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.913292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.913294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.913388 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.913742 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.913748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.913752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.913998 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.914163 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.914405 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.914550 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":88,"threads_started":1}
I20260806 15:54:29.912478 23823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:29.916386 23823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:29.918170 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.894772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.13.223.61:39527)
0806 15:54:29.899806 (+ 5034us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.899810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.899824 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:29.899858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.899861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.899904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.899987 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:29.909171 (+ 9184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.909896 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.912526 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.912990 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.913113 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.914752 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:29.914757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:29.914760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:29.914787 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:29.916277 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.916799 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.916803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.916805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.916894 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:29.917181 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:29.917184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:29.917186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:29.917362 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:29.917514 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.917734 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.917860 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4957,"thread_start_us":103,"threads_started":1}
I20260806 15:54:29.919011 24703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: NON_VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: true } } }
I20260806 15:54:29.919162 24703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.923123 25805 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: attempting to promote NON_VOTER 2ad34f7a3f0b4b3f9ca3d16660b96d3b to VOTER
I20260806 15:54:29.923522 25805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } }
I20260806 15:54:29.924774 24778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Refusing update from remote peer 38e98a7e487340bab964f5dfbbd35b70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 15:54:29.925146 25817 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 15:54:29.926740 25804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.926966 25071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 LEARNER]: Refusing update from remote peer 38e98a7e487340bab964f5dfbbd35b70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 15:54:29.927129 24778 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.927459 25817 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 15:54:29.928573 25804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.928759 25804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.928894 25071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.929159 24703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.929296 24703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.930677 25602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.930795 25602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:29.931406 25819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.931550 25819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:29.931957 25817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38e98a7e487340bab964f5dfbbd35b70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:29.932081 25817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:29.940316 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.934773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:34903 (local address 127.0.0.1:34666)
0806 15:54:29.934960 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.934978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.935068 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.936136 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.936140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.936158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.936285 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.936292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.937772 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.937776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.938806 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.938815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.939926 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.939930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.940017 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.940039 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.940103 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.940180 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":23}
I20260806 15:54:29.940419 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.935096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.13.223.2:34903)
0806 15:54:29.935609 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.935625 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.935653 (+ 28us) server_negotiation.cc:407] Connection header received
0806 15:54:29.935741 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.935748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.935972 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.936171 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.936400 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.937655 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.938928 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.939788 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.939928 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.940110 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.940162 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.940251 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":151,"threads_started":1}
I20260806 15:54:29.941562 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.935111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40601 (local address 127.0.0.1:50402)
0806 15:54:29.935508 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.935524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.935626 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.936979 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.936982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.937001 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.937090 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.937097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.938440 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.938444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.940006 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.940015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.941135 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.941138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.941249 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.941274 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.941361 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.941428 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":96,"thread_start_us":138,"threads_started":1}
I20260806 15:54:29.941673 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.935530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.13.223.3:40601)
0806 15:54:29.936521 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.936546 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.936569 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:54:29.936649 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.936657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.936838 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.937004 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.937182 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.938316 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.940137 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.940988 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.941125 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.941292 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.941392 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.941486 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":151,"threads_started":1}
I20260806 15:54:29.942106 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.935453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:35767 (local address 127.0.0.1:43812)
0806 15:54:29.936445 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.936458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.936608 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.937202 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.937205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.937222 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.937301 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.937306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.938478 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.938483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.939857 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.939869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.941437 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.941441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.941522 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.941545 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.941907 (+ 362us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.941986 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
I20260806 15:54:29.942696 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.935829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.13.223.1:35767)
0806 15:54:29.936099 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.936116 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.936599 (+ 483us) server_negotiation.cc:407] Connection header received
0806 15:54:29.936719 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.936726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.936918 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.937062 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.937415 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.938335 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.939997 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.941143 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.941252 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.941375 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.942412 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.942518 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260806 15:54:29.943615 25453 tablet_service.cc:1467] Processing CreateTablet for tablet c249180112484779b621c5b6060ae74e (DEFAULT_TABLE table=test-table [id=8cd8f1b984084acaa14decb2e16f2b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:29.944373 25672 tablet_service.cc:1467] Processing CreateTablet for tablet c249180112484779b621c5b6060ae74e (DEFAULT_TABLE table=test-table [id=8cd8f1b984084acaa14decb2e16f2b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:29.944917 25453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c249180112484779b621c5b6060ae74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:29.945325 25672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c249180112484779b621c5b6060ae74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:29.945595 25243 tablet_service.cc:1467] Processing CreateTablet for tablet c249180112484779b621c5b6060ae74e (DEFAULT_TABLE table=test-table [id=8cd8f1b984084acaa14decb2e16f2b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:29.946568 25243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c249180112484779b621c5b6060ae74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:29.954748 25827 tablet_bootstrap.cc:492] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Bootstrap starting.
I20260806 15:54:29.956339 25829 tablet_bootstrap.cc:492] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Bootstrap starting.
I20260806 15:54:29.957118 25830 tablet_bootstrap.cc:492] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Bootstrap starting.
I20260806 15:54:29.957623 25827 tablet_bootstrap.cc:654] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:29.958741 25827 log.cc:824] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:29.959026 25829 tablet_bootstrap.cc:654] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:29.959111 25830 tablet_bootstrap.cc:654] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:29.960044 25830 log.cc:824] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:29.960139 25829 log.cc:824] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:29.961251 25827 tablet_bootstrap.cc:492] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: No bootstrap required, opened a new log
I20260806 15:54:29.961472 25827 ts_tablet_manager.cc:1397] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260806 15:54:29.962190 25829 tablet_bootstrap.cc:492] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: No bootstrap required, opened a new log
I20260806 15:54:29.962190 25830 tablet_bootstrap.cc:492] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: No bootstrap required, opened a new log
I20260806 15:54:29.962368 25829 ts_tablet_manager.cc:1397] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260806 15:54:29.962368 25830 ts_tablet_manager.cc:1397] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 15:54:29.965472 25827 raft_consensus.cc:348] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.965852 25827 raft_consensus.cc:374] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:29.965910 25827 raft_consensus.cc:729] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74caae195ad48cabe9946b38b7afe6b, State: Initialized, Role: FOLLOWER
I20260806 15:54:29.966419 25827 consensus_queue.cc:260] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.967018 25830 raft_consensus.cc:348] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.967283 25830 raft_consensus.cc:374] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:29.967343 25830 raft_consensus.cc:729] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c9f851accc49cd900e4b1c97970ace, State: Initialized, Role: FOLLOWER
I20260806 15:54:29.967579 25827 ts_tablet_manager.cc:1428] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 15:54:29.967793 25774 heartbeater.cc:503] Master 127.13.223.62:44803 was elected leader, sending a full tablet report...
I20260806 15:54:29.967782 25829 raft_consensus.cc:348] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.968030 25829 raft_consensus.cc:374] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:29.968086 25829 raft_consensus.cc:729] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b418ae20483d4f17aab8b7620bd13f02, State: Initialized, Role: FOLLOWER
I20260806 15:54:29.968308 25830 consensus_queue.cc:260] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.969336 25830 ts_tablet_manager.cc:1428] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20260806 15:54:29.970459 25350 heartbeater.cc:503] Master 127.13.223.62:44803 was elected leader, sending a full tablet report...
I20260806 15:54:29.972105 25829 consensus_queue.cc:260] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.973318 25829 ts_tablet_manager.cc:1428] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20260806 15:54:29.973528 25560 heartbeater.cc:503] Master 127.13.223.62:44803 was elected leader, sending a full tablet report...
I20260806 15:54:29.973889 25838 raft_consensus.cc:482] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:29.974035 25838 raft_consensus.cc:504] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.976883 25838 leader_election.cc:288] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601), b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903)
I20260806 15:54:29.980276 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.976210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:33363 (local address 127.13.223.3:40601)
0806 15:54:29.976328 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.976332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.976346 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:29.976389 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.976392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.976445 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.976556 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.976944 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.977724 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.979287 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.979675 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.979806 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.979859 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.980090 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.980162 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:54:29.980314 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.975887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40601 (local address 127.13.223.1:33363)
0806 15:54:29.976172 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.976189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.976302 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.976611 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.976614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.976631 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.976764 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.976771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.977841 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.977843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.979168 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.979177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.979831 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.979834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.979924 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.979952 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.980055 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.980144 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":137,"threads_started":1}
I20260806 15:54:29.980901 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.976616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:49635 (local address 127.13.223.2:34903)
0806 15:54:29.976851 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:29.976855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:29.977224 (+ 369us) server_negotiation.cc:407] Connection header received
0806 15:54:29.977411 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:29.977414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:29.977458 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:29.977509 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:29.977973 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.978712 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.979734 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.980135 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.980265 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.980311 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:29.980724 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 15:54:29.980795 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260806 15:54:29.980939 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:29.976547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:34903 (local address 127.13.223.1:49635)
0806 15:54:29.977127 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:29.977143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:29.977289 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:29.977645 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:29.977649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:29.977667 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:29.977845 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.977854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.978828 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.978830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.979615 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:29.979623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:29.980259 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:29.980263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:29.980591 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:29.980611 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 15:54:29.980695 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 15:54:29.980764 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":136,"threads_started":1}
I20260806 15:54:29.981030 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c249180112484779b621c5b6060ae74e" candidate_uuid: "95c9f851accc49cd900e4b1c97970ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74caae195ad48cabe9946b38b7afe6b" is_pre_election: true
I20260806 15:54:29.981345 25698 raft_consensus.cc:2454] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95c9f851accc49cd900e4b1c97970ace in term 0.
I20260806 15:54:29.981784 25490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c249180112484779b621c5b6060ae74e" candidate_uuid: "95c9f851accc49cd900e4b1c97970ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b418ae20483d4f17aab8b7620bd13f02" is_pre_election: true
I20260806 15:54:29.981855 25167 leader_election.cc:302] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95c9f851accc49cd900e4b1c97970ace, a74caae195ad48cabe9946b38b7afe6b; no voters:
I20260806 15:54:29.982079 25490 raft_consensus.cc:2454] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95c9f851accc49cd900e4b1c97970ace in term 0.
I20260806 15:54:29.982168 25838 raft_consensus.cc:2781] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:29.982256 25838 raft_consensus.cc:482] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:29.982308 25838 raft_consensus.cc:3037] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:29.983485 25838 raft_consensus.cc:504] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.983911 25838 leader_election.cc:288] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [CANDIDATE]: Term 1 election: Requested vote from peers a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601), b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903)
I20260806 15:54:29.984195 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c249180112484779b621c5b6060ae74e" candidate_uuid: "95c9f851accc49cd900e4b1c97970ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74caae195ad48cabe9946b38b7afe6b"
I20260806 15:54:29.984308 25490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c249180112484779b621c5b6060ae74e" candidate_uuid: "95c9f851accc49cd900e4b1c97970ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b418ae20483d4f17aab8b7620bd13f02"
I20260806 15:54:29.984344 25698 raft_consensus.cc:3037] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:29.984422 25490 raft_consensus.cc:3037] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:29.985915 25698 raft_consensus.cc:2454] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95c9f851accc49cd900e4b1c97970ace in term 1.
I20260806 15:54:29.985915 25490 raft_consensus.cc:2454] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95c9f851accc49cd900e4b1c97970ace in term 1.
I20260806 15:54:29.986205 25172 leader_election.cc:302] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95c9f851accc49cd900e4b1c97970ace, b418ae20483d4f17aab8b7620bd13f02; no voters:
I20260806 15:54:29.986375 25838 raft_consensus.cc:2781] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:29.986513 25838 raft_consensus.cc:686] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 1 LEADER]: Becoming Leader. State: Replica: 95c9f851accc49cd900e4b1c97970ace, State: Running, Role: LEADER
I20260806 15:54:29.986877 25838 consensus_queue.cc:237] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:29.991271 23822 catalog_manager.cc:5495] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace reported cstate change: term changed from 0 to 1, leader changed from <none> to 95c9f851accc49cd900e4b1c97970ace (127.13.223.1). New cstate: current_term: 1 leader_uuid: "95c9f851accc49cd900e4b1c97970ace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } health_report { overall_health: HEALTHY } } }
I20260806 15:54:30.019747 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:30.016007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.13.223.1:35767)
0806 15:54:30.016143 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:30.016147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:30.016167 (+ 20us) server_negotiation.cc:407] Connection header received
0806 15:54:30.016353 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:30.016356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:30.016414 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:30.016582 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:30.016871 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.017380 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.018201 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.018422 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.018500 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:30.018770 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:30.019357 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:30.019481 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:30.019542 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 15:54:30.019604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:54:30.026170 25698 raft_consensus.cc:1261] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Refusing update from remote peer 95c9f851accc49cd900e4b1c97970ace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:30.026172 25490 raft_consensus.cc:1261] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Refusing update from remote peer 95c9f851accc49cd900e4b1c97970ace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:30.027009 25839 consensus_queue.cc:1035] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:30.027194 25838 consensus_queue.cc:1035] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [LEADER]: Connected to new peer: Peer: permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:30.036952 25851 mvcc.cc:205] Tried to move back new op lower bound from 7315585720419414016 to 7315585720269348864. Current Snapshot: MvccSnapshot[applied={T|T < 7315585720419414016}]
I20260806 15:54:30.040452 25852 mvcc.cc:205] Tried to move back new op lower bound from 7315585720419414016 to 7315585720269348864. Current Snapshot: MvccSnapshot[applied={T|T < 7315585720419414016}]
I20260806 15:54:30.045465 25847 mvcc.cc:205] Tried to move back new op lower bound from 7315585720419414016 to 7315585720269348864. Current Snapshot: MvccSnapshot[applied={T|T < 7315585720419414016 or (T in {7315585720419414016})}]
I20260806 15:54:30.048682 23822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:30.048875 23822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:30.053265 23822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38e98a7e487340bab964f5dfbbd35b70: Sending DeleteTablet for 3 replicas of tablet c249180112484779b621c5b6060ae74e
I20260806 15:54:30.054136 25453 tablet_service.cc:1514] Processing DeleteTablet for tablet c249180112484779b621c5b6060ae74e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:30 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:34666
I20260806 15:54:30.054299 25672 tablet_service.cc:1514] Processing DeleteTablet for tablet c249180112484779b621c5b6060ae74e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:30 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:50402
I20260806 15:54:30.054410 25243 tablet_service.cc:1514] Processing DeleteTablet for tablet c249180112484779b621c5b6060ae74e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:30 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:43812
I20260806 15:54:30.054848 25863 tablet_replica.cc:330] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: stopping tablet replica
I20260806 15:54:30.055225 25863 raft_consensus.cc:2229] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:30.055589 25863 raft_consensus.cc:2258] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:30.056584 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 23771
I20260806 15:54:30.064999 25863 ts_tablet_manager.cc:1905] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:30.065028 25864 tablet_replica.cc:330] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: stopping tablet replica
I20260806 15:54:30.065405 25864 raft_consensus.cc:2229] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:30.065776 25864 raft_consensus.cc:2258] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:30.066339 25865 tablet_replica.cc:330] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: stopping tablet replica
I20260806 15:54:30.066687 25865 raft_consensus.cc:2229] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:30.067127 25865 raft_consensus.cc:2258] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:30.068325 25863 ts_tablet_manager.cc:1918] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:30.068434 25863 log.cc:1196] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/c249180112484779b621c5b6060ae74e
I20260806 15:54:30.068763 25865 ts_tablet_manager.cc:1905] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:30.068840 25863 ts_tablet_manager.cc:1939] T c249180112484779b621c5b6060ae74e P a74caae195ad48cabe9946b38b7afe6b: Deleting consensus metadata
I20260806 15:54:30.072360 25865 ts_tablet_manager.cc:1918] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:30.072476 25865 log.cc:1196] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/c249180112484779b621c5b6060ae74e
I20260806 15:54:30.072860 25865 ts_tablet_manager.cc:1939] T c249180112484779b621c5b6060ae74e P 95c9f851accc49cd900e4b1c97970ace: Deleting consensus metadata
W20260806 15:54:30.074283 25172 connection.cc:535] server connection from 127.0.0.1:43812 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 15:54:30.074337 25597 connection.cc:535] server connection from 127.0.0.1:50402 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 15:54:30.075855 25864 ts_tablet_manager.cc:1905] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:30.079197 25864 ts_tablet_manager.cc:1918] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:30.079300 25864 log.cc:1196] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/c249180112484779b621c5b6060ae74e
I20260806 15:54:30.079653 25864 ts_tablet_manager.cc:1939] T c249180112484779b621c5b6060ae74e P b418ae20483d4f17aab8b7620bd13f02: Deleting consensus metadata
W20260806 15:54:30.081034 25391 connection.cc:441] server connection from 127.0.0.1:34666 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34666 (request call id 1) could send its response
I20260806 15:54:30.082989 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:30.056747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.13.223.61:39527)
0806 15:54:30.064083 (+ 7336us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:30.064090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:30.068170 (+ 4080us) server_negotiation.cc:407] Connection header received
0806 15:54:30.068420 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:30.068424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:30.068516 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:30.068583 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:30.072695 (+ 4112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.073497 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.074809 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.075156 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.075270 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:30.077520 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:30.077524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:30.077527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:30.077560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:30.079462 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.081252 (+ 1790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.081256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.081258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.081345 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.081879 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.081886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.081889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.082131 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:30.082258 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:30.082690 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 15:54:30.082829 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7214}
I20260806 15:54:30.091615 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:30.079908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.13.223.61:39527)
0806 15:54:30.080122 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:30.080126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:30.080991 (+ 865us) server_negotiation.cc:407] Connection header received
0806 15:54:30.081036 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:30.081039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:30.081086 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:30.081160 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:30.082861 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.083558 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.084991 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.085376 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.085466 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:30.087188 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:30.087192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:30.087195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:30.087220 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:30.089478 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.089984 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.089989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.089992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.090081 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.090445 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.090451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.090456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.090713 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:30.090833 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:30.091310 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 15:54:30.091436 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260806 15:54:30.093698 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:30.079787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.13.223.60:39727)
0806 15:54:30.080036 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:30.080041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:30.080587 (+ 546us) server_negotiation.cc:407] Connection header received
0806 15:54:30.080767 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:30.080770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:30.080822 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:30.080898 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:30.081987 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.082797 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.084013 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.084356 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.084433 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:30.089830 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:30.089834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:30.089837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:30.089867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:30.091856 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.092348 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.092351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.092353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.092442 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.092821 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.092828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.092832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.093062 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:30.093193 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:30.093424 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 15:54:30.093535 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260806 15:54:30.095338 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:30.064620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.13.223.60:39727)
0806 15:54:30.064762 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:30.064767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:30.084365 (+ 19598us) server_negotiation.cc:407] Connection header received
0806 15:54:30.084633 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:30.084636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:30.084686 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:30.084765 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:30.085527 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.086306 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.087450 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:30.087817 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:30.087922 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:30.089658 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:30.089662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:30.089664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:30.089697 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:30.092925 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.094049 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.094051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.094052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.094126 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:30.094449 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:30.094452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:30.094454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:30.094640 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:30.094772 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:30.095058 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 15:54:30.095189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:54:31.040453 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.039796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:46365)
0806 15:54:31.040191 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.040310 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":110,"threads_started":1}
W20260806 15:54:31.040825 25560 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:44803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.044345 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.043921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:60817)
0806 15:54:31.044179 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.044261 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
W20260806 15:54:31.044724 25350 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:44803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.047065 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.046655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:45133)
0806 15:54:31.046869 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.046942 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
W20260806 15:54:31.047394 25774 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:44803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.554286 25881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 38e98a7e487340bab964f5dfbbd35b70)
I20260806 15:54:31.554414 25882 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 38e98a7e487340bab964f5dfbbd35b70)
I20260806 15:54:31.554648 25881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } }
I20260806 15:54:31.554674 25882 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } }
I20260806 15:54:31.555840 25882 leader_election.cc:288] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803), 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60:39727)
I20260806 15:54:31.555866 25881 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803), b59818fcaf734e5b85ce691bda0e822a (127.13.223.61:39527)
I20260806 15:54:31.556313 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.555921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52960)
0806 15:54:31.556134 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.556202 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
W20260806 15:54:31.556517 25035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.556852 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.555936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52954)
0806 15:54:31.556671 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.556744 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":63,"threads_started":1}
W20260806 15:54:31.557128 24747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.557663 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.557313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52962)
0806 15:54:31.557537 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.557588 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260806 15:54:31.557767 25035 leader_election.cc:334] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803): Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.558573 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.558322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52972)
0806 15:54:31.558426 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.558492 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 15:54:31.558756 24747 leader_election.cc:334] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803): Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.561020 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.556729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.13.223.60:39727)
0806 15:54:31.557136 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.557145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.557160 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:31.557209 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.557215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.557311 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.557728 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.557828 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.558558 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.559876 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.560490 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.560660 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.560708 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.560884 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.560939 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":113,"threads_started":1}
I20260806 15:54:31.561056 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.556227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:39727 (local address 127.0.0.1:46234)
0806 15:54:31.556811 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.556824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.556907 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.557451 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.557455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.557476 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.557614 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.557622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.558718 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.558721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.559701 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.559708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.560645 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.560648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.560731 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.560787 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.560864 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.560928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":21,"thread_start_us":67,"threads_started":1}
I20260806 15:54:31.561589 25071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b59818fcaf734e5b85ce691bda0e822a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" is_pre_election: true
I20260806 15:54:31.561690 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.556203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:39527 (local address 127.0.0.1:45368)
0806 15:54:31.556462 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.556475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.556605 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.557496 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.557500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.557520 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.557648 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.557655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.559388 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.559391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.560490 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.560497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.561413 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.561416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.561495 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.561513 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.561560 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.561606 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260806 15:54:31.561820 25071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59818fcaf734e5b85ce691bda0e822a in term 1.
I20260806 15:54:31.562171 24747 leader_election.cc:302] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad34f7a3f0b4b3f9ca3d16660b96d3b, b59818fcaf734e5b85ce691bda0e822a; no voters: 38e98a7e487340bab964f5dfbbd35b70
I20260806 15:54:31.562283 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.556408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.13.223.61:39527)
0806 15:54:31.557174 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.557179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.557194 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:31.557290 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.557294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.557352 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.557955 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.558089 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.559268 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.560628 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.561245 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.561988 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.562030 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.562082 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.562144 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":459,"threads_started":1}
I20260806 15:54:31.562445 25882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 15:54:31.562527 25882 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 38e98a7e487340bab964f5dfbbd35b70)
I20260806 15:54:31.562582 25882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 1 FOLLOWER]: Advancing to term 2
I20260806 15:54:31.562784 24778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b59818fcaf734e5b85ce691bda0e822a" is_pre_election: true
I20260806 15:54:31.563726 25882 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } }
I20260806 15:54:31.564050 24778 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ad34f7a3f0b4b3f9ca3d16660b96d3b in current term 2: Already voted for candidate b59818fcaf734e5b85ce691bda0e822a in this term.
I20260806 15:54:31.564056 25882 leader_election.cc:288] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [CANDIDATE]: Term 2 election: Requested vote from peers 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803), 2ad34f7a3f0b4b3f9ca3d16660b96d3b (127.13.223.60:39727)
I20260806 15:54:31.564435 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.564235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52974)
0806 15:54:31.564344 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.564387 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 15:54:31.565070 25071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b59818fcaf734e5b85ce691bda0e822a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b"
I20260806 15:54:31.565230 25071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 1 FOLLOWER]: Advancing to term 2
I20260806 15:54:31.565296 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.565030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52978)
0806 15:54:31.565161 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.565228 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 15:54:31.565344 25036 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ad34f7a3f0b4b3f9ca3d16660b96d3b; no voters: 38e98a7e487340bab964f5dfbbd35b70, b59818fcaf734e5b85ce691bda0e822a
W20260806 15:54:31.565431 24747 leader_election.cc:334] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803): Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
I20260806 15:54:31.566340 25071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59818fcaf734e5b85ce691bda0e822a in term 2.
I20260806 15:54:31.566450 25881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:54:31.566642 24747 leader_election.cc:302] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad34f7a3f0b4b3f9ca3d16660b96d3b, b59818fcaf734e5b85ce691bda0e822a; no voters: 38e98a7e487340bab964f5dfbbd35b70
I20260806 15:54:31.566836 25882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 15:54:31.567068 25882 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [term 2 LEADER]: Becoming Leader. State: Replica: b59818fcaf734e5b85ce691bda0e822a, State: Running, Role: LEADER
I20260806 15:54:31.567319 25882 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } }
I20260806 15:54:31.568522 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59818fcaf734e5b85ce691bda0e822a. Latest consensus state: current_term: 2 leader_uuid: "b59818fcaf734e5b85ce691bda0e822a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:31.568679 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:31.569027 25902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:31.571349 25902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:31.571758 25902 catalog_manager.cc:1259] Loaded cluster ID: e96f23a66721400d9de599be83617ee0
I20260806 15:54:31.571805 25902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:31.572336 25902 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:31.572804 25902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Loaded TSK: 0
I20260806 15:54:31.573499 25902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:31.617602 24765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:31.620671 24765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:31.623606 25071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [term 2 FOLLOWER]: Refusing update from remote peer b59818fcaf734e5b85ce691bda0e822a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 15:54:31.623631 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.623331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:52998)
0806 15:54:31.623506 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.623555 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 15:54:31.624168 25900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 15:54:31.624437 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.624214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.0.0.1:53002)
0806 15:54:31.624321 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.624372 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 15:54:31.624575 24747 consensus_peers.cc:487] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a -> Peer 38e98a7e487340bab964f5dfbbd35b70 (127.13.223.62:44803): Couldn't send request to peer 38e98a7e487340bab964f5dfbbd35b70. Status: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 15:54:31.625656 25881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59818fcaf734e5b85ce691bda0e822a. Latest consensus state: current_term: 2 leader_uuid: "b59818fcaf734e5b85ce691bda0e822a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:31.625765 25881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:31.626962 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b59818fcaf734e5b85ce691bda0e822a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:31.627079 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:31.627336 25881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b59818fcaf734e5b85ce691bda0e822a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "38e98a7e487340bab964f5dfbbd35b70" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44803 } } peers { permanent_uuid: "b59818fcaf734e5b85ce691bda0e822a" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 39527 } attrs { promote: false } } peers { permanent_uuid: "2ad34f7a3f0b4b3f9ca3d16660b96d3b" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 39727 } attrs { promote: false } } }
I20260806 15:54:31.627453 25881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad34f7a3f0b4b3f9ca3d16660b96d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:54:31.637866 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.633185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40601 (local address 127.0.0.1:50416)
0806 15:54:31.633546 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.633561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.633703 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.633984 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.633987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.634005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.634135 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.634142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.635301 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.635305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.636603 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.636613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.637480 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.637484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.637578 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.637609 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.637657 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.637725 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":19}
I20260806 15:54:31.638270 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.633757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:35767 (local address 127.0.0.1:43828)
0806 15:54:31.634434 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.634448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.634506 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.634765 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.634771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.634788 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.634874 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.634879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.635689 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.635692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.636896 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.636904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.637911 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.637918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.638017 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.638041 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.638089 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.638157 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260806 15:54:31.638370 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.633450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.13.223.3:40601)
0806 15:54:31.633704 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.633709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.633725 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:31.633782 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.633785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.633832 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.634242 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.634374 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.635174 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.636740 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.637353 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.638050 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.638138 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.638173 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.638241 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
I20260806 15:54:31.638914 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.634190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.13.223.1:35767)
0806 15:54:31.634465 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.634469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.634483 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:31.634588 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.634591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.634635 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.634738 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.634993 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.635543 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.637162 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.637783 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.638647 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.638713 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.638755 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.638822 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260806 15:54:31.638903 25672 tablet_service.cc:1467] Processing CreateTablet for tablet a917d835a9cc4a8399c89ac2272dc0af (DEFAULT_TABLE table=test-table [id=ceee3f6053564fb894c98108fe3b4308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:31.639004 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.633402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:34903 (local address 127.0.0.1:34668)
0806 15:54:31.634012 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.634024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.634157 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.634565 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.634567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.634584 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.634676 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.634699 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.635699 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.635702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.637515 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.637525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.638333 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.638605 (+ 272us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.638712 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.638736 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.638824 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.638885 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":23}
I20260806 15:54:31.639214 25672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a917d835a9cc4a8399c89ac2272dc0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:31.639343 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.633526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.13.223.2:34903)
0806 15:54:31.633747 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.633754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.634100 (+ 346us) server_negotiation.cc:407] Connection header received
0806 15:54:31.634288 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.634294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.634354 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.634440 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.634861 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.635602 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.637699 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.638220 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.638332 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.638400 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.639139 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.639225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260806 15:54:31.639509 25243 tablet_service.cc:1467] Processing CreateTablet for tablet a917d835a9cc4a8399c89ac2272dc0af (DEFAULT_TABLE table=test-table [id=ceee3f6053564fb894c98108fe3b4308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:31.639827 25243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a917d835a9cc4a8399c89ac2272dc0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:31.639886 25453 tablet_service.cc:1467] Processing CreateTablet for tablet a917d835a9cc4a8399c89ac2272dc0af (DEFAULT_TABLE table=test-table [id=ceee3f6053564fb894c98108fe3b4308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:31.640142 25453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a917d835a9cc4a8399c89ac2272dc0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:31.641860 25920 tablet_bootstrap.cc:492] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Bootstrap starting.
I20260806 15:54:31.642565 25921 tablet_bootstrap.cc:492] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: Bootstrap starting.
I20260806 15:54:31.642911 25920 tablet_bootstrap.cc:654] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:31.643065 25922 tablet_bootstrap.cc:492] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Bootstrap starting.
I20260806 15:54:31.643410 25921 tablet_bootstrap.cc:654] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:31.644006 25922 tablet_bootstrap.cc:654] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:31.644312 25920 tablet_bootstrap.cc:492] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: No bootstrap required, opened a new log
I20260806 15:54:31.644450 25920 ts_tablet_manager.cc:1397] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:31.644651 25921 tablet_bootstrap.cc:492] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: No bootstrap required, opened a new log
I20260806 15:54:31.644807 25921 ts_tablet_manager.cc:1397] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 15:54:31.645103 25920 raft_consensus.cc:348] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.645220 25920 raft_consensus.cc:374] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:31.645207 25921 raft_consensus.cc:348] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.645256 25920 raft_consensus.cc:729] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74caae195ad48cabe9946b38b7afe6b, State: Initialized, Role: FOLLOWER
I20260806 15:54:31.645289 25921 raft_consensus.cc:374] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:31.645310 25921 raft_consensus.cc:729] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c9f851accc49cd900e4b1c97970ace, State: Initialized, Role: FOLLOWER
I20260806 15:54:31.645377 25922 tablet_bootstrap.cc:492] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: No bootstrap required, opened a new log
I20260806 15:54:31.645385 25920 consensus_queue.cc:260] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.645453 25922 ts_tablet_manager.cc:1397] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:31.645449 25921 consensus_queue.cc:260] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.645730 25920 ts_tablet_manager.cc:1428] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 15:54:31.645884 25921 ts_tablet_manager.cc:1428] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 15:54:31.645882 25922 raft_consensus.cc:348] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.646009 25922 raft_consensus.cc:374] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:31.646055 25922 raft_consensus.cc:729] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b418ae20483d4f17aab8b7620bd13f02, State: Initialized, Role: FOLLOWER
I20260806 15:54:31.646202 25922 consensus_queue.cc:260] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.646574 25922 ts_tablet_manager.cc:1428] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:31.649187 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.646657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:57257)
0806 15:54:31.648977 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.649064 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2213,"thread_start_us":91,"threads_started":1}
I20260806 15:54:31.649526 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.646242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:54993)
0806 15:54:31.649367 (+ 3125us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.649423 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3061,"thread_start_us":99,"threads_started":1}
I20260806 15:54:31.649933 25777 heartbeater.cc:503] Master 127.13.223.61:39527 was elected leader, sending a full tablet report...
I20260806 15:54:31.650100 25562 heartbeater.cc:503] Master 127.13.223.61:39527 was elected leader, sending a full tablet report...
I20260806 15:54:31.650372 25352 heartbeater.cc:503] Master 127.13.223.61:39527 was elected leader, sending a full tablet report...
I20260806 15:54:31.650841 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.650352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:45959)
0806 15:54:31.650716 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.650768 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260806 15:54:31.651700 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.651051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:33735)
0806 15:54:31.651578 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.651636 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457}
I20260806 15:54:31.653051 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.647391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:40129)
0806 15:54:31.652892 (+ 5501us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.652952 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5433,"thread_start_us":97,"threads_started":1}
I20260806 15:54:31.653831 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.653585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:40895)
0806 15:54:31.653703 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.653751 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 15:54:31.657361 25563 tablet.cc:2369] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:31.686896 25354 tablet.cc:2369] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:31.885874 25778 tablet.cc:2369] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:31.977041 25927 raft_consensus.cc:482] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:31.977224 25927 raft_consensus.cc:504] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.978436 25927 leader_election.cc:288] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903), 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767)
I20260806 15:54:31.982254 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.978627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:34903 (local address 127.13.223.3:40185)
0806 15:54:31.978801 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.978818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.979166 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.979448 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.979450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.979469 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.979575 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.979580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.980304 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.980307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.981171 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.981180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.981836 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.981840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.981949 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.981967 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.982053 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.982121 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 15:54:31.982254 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.978801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:40185 (local address 127.13.223.2:34903)
0806 15:54:31.978958 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.978963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.979112 (+ 149us) server_negotiation.cc:407] Connection header received
0806 15:54:31.979305 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.979308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.979348 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.979434 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.979685 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.980173 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.981295 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.981685 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.981843 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.981894 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.982069 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.982142 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:54:31.982494 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.979088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:58973 (local address 127.13.223.1:35767)
0806 15:54:31.979286 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:31.979298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:31.979361 (+ 63us) server_negotiation.cc:407] Connection header received
0806 15:54:31.979543 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:31.979545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:31.979585 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:31.979666 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:31.979907 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.980408 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.981418 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.981920 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.982057 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.982109 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:31.982280 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 15:54:31.982363 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:54:31.982697 25490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a917d835a9cc4a8399c89ac2272dc0af" candidate_uuid: "a74caae195ad48cabe9946b38b7afe6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b418ae20483d4f17aab8b7620bd13f02" is_pre_election: true
I20260806 15:54:31.982758 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.978799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:35767 (local address 127.13.223.3:58973)
0806 15:54:31.979144 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.979161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:31.979410 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:31.979678 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:31.979681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:31.979696 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:31.979783 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.979789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.980543 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.980549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.981287 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:31.981296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:31.982057 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:31.982060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:31.982148 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:31.982169 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:54:31.982263 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 15:54:31.982623 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
I20260806 15:54:31.982851 25490 raft_consensus.cc:2454] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74caae195ad48cabe9946b38b7afe6b in term 0.
I20260806 15:54:31.983271 25599 leader_election.cc:302] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74caae195ad48cabe9946b38b7afe6b, b418ae20483d4f17aab8b7620bd13f02; no voters:
I20260806 15:54:31.983336 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a917d835a9cc4a8399c89ac2272dc0af" candidate_uuid: "a74caae195ad48cabe9946b38b7afe6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c9f851accc49cd900e4b1c97970ace" is_pre_election: true
I20260806 15:54:31.983592 25927 raft_consensus.cc:2781] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:31.983662 25927 raft_consensus.cc:482] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:31.983639 25277 raft_consensus.cc:2454] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74caae195ad48cabe9946b38b7afe6b in term 0.
I20260806 15:54:31.983716 25927 raft_consensus.cc:3037] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:31.985096 25927 raft_consensus.cc:504] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.985435 25927 leader_election.cc:288] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [CANDIDATE]: Term 1 election: Requested vote from peers b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903), 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767)
I20260806 15:54:31.985791 25490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a917d835a9cc4a8399c89ac2272dc0af" candidate_uuid: "a74caae195ad48cabe9946b38b7afe6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b418ae20483d4f17aab8b7620bd13f02"
I20260806 15:54:31.985819 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a917d835a9cc4a8399c89ac2272dc0af" candidate_uuid: "a74caae195ad48cabe9946b38b7afe6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c9f851accc49cd900e4b1c97970ace"
I20260806 15:54:31.985934 25490 raft_consensus.cc:3037] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:31.985956 25277 raft_consensus.cc:3037] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:31.986809 25490 raft_consensus.cc:2454] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74caae195ad48cabe9946b38b7afe6b in term 1.
I20260806 15:54:31.986907 25277 raft_consensus.cc:2454] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74caae195ad48cabe9946b38b7afe6b in term 1.
I20260806 15:54:31.987130 25599 leader_election.cc:302] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74caae195ad48cabe9946b38b7afe6b, b418ae20483d4f17aab8b7620bd13f02; no voters:
I20260806 15:54:31.987309 25927 raft_consensus.cc:2781] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:31.987546 25927 raft_consensus.cc:686] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 1 LEADER]: Becoming Leader. State: Replica: a74caae195ad48cabe9946b38b7afe6b, State: Running, Role: LEADER
I20260806 15:54:31.988075 25927 consensus_queue.cc:237] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } }
I20260806 15:54:31.988318 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:31.988009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:54501)
0806 15:54:31.988130 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:31.988182 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 15:54:31.990631 24763 catalog_manager.cc:5495] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b reported cstate change: term changed from 0 to 1, leader changed from <none> to a74caae195ad48cabe9946b38b7afe6b (127.13.223.3). New cstate: current_term: 1 leader_uuid: "a74caae195ad48cabe9946b38b7afe6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74caae195ad48cabe9946b38b7afe6b" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:32.005335 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.001564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.13.223.3:40601)
0806 15:54:32.001711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:32.001717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:32.001736 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:54:32.001951 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:32.001954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:32.002005 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:32.002120 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:32.002374 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:32.002889 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:32.003803 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:32.004025 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:32.004134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:32.004418 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:32.004929 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:32.005055 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:32.005142 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 15:54:32.005197 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 15:54:32.008219 25490 raft_consensus.cc:1261] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Refusing update from remote peer a74caae195ad48cabe9946b38b7afe6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:32.008468 25277 raft_consensus.cc:1261] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Refusing update from remote peer a74caae195ad48cabe9946b38b7afe6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:32.008771 25927 consensus_queue.cc:1035] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b418ae20483d4f17aab8b7620bd13f02" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:32.009181 25943 consensus_queue.cc:1035] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95c9f851accc49cd900e4b1c97970ace" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:32.010970 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.010653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:36443)
0806 15:54:32.010817 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.010885 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 15:54:32.011246 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.010381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:60379)
0806 15:54:32.011094 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.011164 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":632}
I20260806 15:54:32.015547 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.015264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.1:46975)
0806 15:54:32.015410 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.015467 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 15:54:32.016409 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.013903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:35791)
0806 15:54:32.016284 (+ 2381us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.016340 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2306}
I20260806 15:54:32.017409 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.017155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.2:46635)
0806 15:54:32.017291 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.017341 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 15:54:32.017422 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.017133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44803 (local address 127.13.223.3:36343)
0806 15:54:32.017323 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.017368 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260806 15:54:32.023311 24763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:32.023454 24763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:32.027519 24763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b59818fcaf734e5b85ce691bda0e822a: Sending DeleteTablet for 3 replicas of tablet a917d835a9cc4a8399c89ac2272dc0af
I20260806 15:54:32.028278 25453 tablet_service.cc:1514] Processing DeleteTablet for tablet a917d835a9cc4a8399c89ac2272dc0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:32 UTC) from {username='oryx', principal='oryx/127.13.223.61@KRBTEST.COM'} at 127.0.0.1:34668
I20260806 15:54:32.028620 25243 tablet_service.cc:1514] Processing DeleteTablet for tablet a917d835a9cc4a8399c89ac2272dc0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:32 UTC) from {username='oryx', principal='oryx/127.13.223.61@KRBTEST.COM'} at 127.0.0.1:43828
I20260806 15:54:32.028640 25672 tablet_service.cc:1514] Processing DeleteTablet for tablet a917d835a9cc4a8399c89ac2272dc0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:32 UTC) from {username='oryx', principal='oryx/127.13.223.61@KRBTEST.COM'} at 127.0.0.1:50416
I20260806 15:54:32.029022 25963 tablet_replica.cc:330] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: stopping tablet replica
I20260806 15:54:32.029042 25967 tablet_replica.cc:330] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: stopping tablet replica
I20260806 15:54:32.029156 25967 raft_consensus.cc:2229] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:32.029156 25963 raft_consensus.cc:2229] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:32.029306 25963 raft_consensus.cc:2258] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:32.029443 25965 tablet_replica.cc:330] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace: stopping tablet replica
I20260806 15:54:32.029567 25965 raft_consensus.cc:2229] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:32.029722 25965 raft_consensus.cc:2258] T a917d835a9cc4a8399c89ac2272dc0af P 95c9f851accc49cd900e4b1c97970ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:32.030071 25967 raft_consensus.cc:2258] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:32.030119 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 24991
I20260806 15:54:32.030216 25963 ts_tablet_manager.cc:1905] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:32.031225 25967 ts_tablet_manager.cc:1905] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:32.033159 25963 ts_tablet_manager.cc:1918] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:32.033231 25963 log.cc:1196] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/a917d835a9cc4a8399c89ac2272dc0af
I20260806 15:54:32.033516 25963 ts_tablet_manager.cc:1939] T a917d835a9cc4a8399c89ac2272dc0af P b418ae20483d4f17aab8b7620bd13f02: Deleting consensus metadata
I20260806 15:54:32.034061 25967 ts_tablet_manager.cc:1918] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:32.034135 25967 log.cc:1196] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/a917d835a9cc4a8399c89ac2272dc0af
I20260806 15:54:32.034297 24748 catalog_manager.cc:4841] TS b418ae20483d4f17aab8b7620bd13f02 (127.13.223.2:34903): tablet a917d835a9cc4a8399c89ac2272dc0af (table test-table [id=ceee3f6053564fb894c98108fe3b4308]) successfully deleted
I20260806 15:54:32.034431 25967 ts_tablet_manager.cc:1939] T a917d835a9cc4a8399c89ac2272dc0af P a74caae195ad48cabe9946b38b7afe6b: Deleting consensus metadata
I20260806 15:54:32.035038 24744 catalog_manager.cc:4841] TS a74caae195ad48cabe9946b38b7afe6b (127.13.223.3:40601): tablet a917d835a9cc4a8399c89ac2272dc0af (table test-table [id=ceee3f6053564fb894c98108fe3b4308]) successfully deleted
I20260806 15:54:32.046856 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 25363
I20260806 15:54:32.047434 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:32.047073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:35767 (local address 127.0.0.1:43840)
0806 15:54:32.047275 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:32.047331 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260806 15:54:32.047686 24744 catalog_manager.cc:4584] TS 95c9f851accc49cd900e4b1c97970ace (127.13.223.1:35767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a917d835a9cc4a8399c89ac2272dc0af: Network error: Client connection negotiation failed: client connection to 127.13.223.1:35767: connect: Connection refused (error 111)
I20260806 15:54:32.063642 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 25568
I20260806 15:54:32.077658 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 24547
I20260806 15:54:32.089731 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 24724
2026-08-06T15:54:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5956 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:32 dist-test-slave-rwrg krb5kdc[25971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:32 dist-test-slave-rwrg krb5kdc[25971](info): set up 2 sockets
Aug 06 15:54:32 dist-test-slave-rwrg krb5kdc[25971](info): commencing operation
krb5kdc: starting...
W20260806 15:54:33.358420 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:33 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab.
Entry for principal oryx/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab.
I20260806 15:54:33.491473 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34829
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab
--principal=oryx/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:33.612242 25989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:33.612612 25989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:33.612660 25989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:33.616766 25989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:33.616832 25989 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 15:54:33.616848 25989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:33.616868 25989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:33.616883 25989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:33.616940 25989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:33.622502 25989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.62.keytab
--principal=oryx/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:33.623626 25989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:33.624684 25989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:33.634054 25999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:33.634091 25997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:33.634054 25996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:33.634419 25989 server_base.cc:1034] running on GCE node
I20260806 15:54:33.635080 25989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:33.636268 25989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:33.637471 25989 hybrid_clock.cc:648] HybridClock initialized: now 1786031673637457 us; error 35 us; skew 500 ppm
Aug 06 15:54:33 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031673, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:33.640812 25989 init.cc:377] Logged in from keytab as oryx/127.13.223.62@KRBTEST.COM (short username oryx)
I20260806 15:54:33.641811 25989 webserver.cc:460] Webserver started at http://127.13.223.62:33433/ using document root <none> and password file <none>
I20260806 15:54:33.642445 25989 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:33.642547 25989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:33.642779 25989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:33.645071 25989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "e32a946d5a3a4a74a93d2c47501415b0"
format_stamp: "Formatted at 2026-08-06 15:54:33 on dist-test-slave-rwrg"
server_key: "acae38ea2d9988f389c652a342d68872"
server_key_iv: "fafc588ae0c64958c5ea625d238b1b01"
server_key_version: "encryptionkey@0"
I20260806 15:54:33.645670 25989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "e32a946d5a3a4a74a93d2c47501415b0"
format_stamp: "Formatted at 2026-08-06 15:54:33 on dist-test-slave-rwrg"
server_key: "acae38ea2d9988f389c652a342d68872"
server_key_iv: "fafc588ae0c64958c5ea625d238b1b01"
server_key_version: "encryptionkey@0"
I20260806 15:54:33.649778 25989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 15:54:33.652349 26006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:33.653698 25989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:54:33.653849 25989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "e32a946d5a3a4a74a93d2c47501415b0"
format_stamp: "Formatted at 2026-08-06 15:54:33 on dist-test-slave-rwrg"
server_key: "acae38ea2d9988f389c652a342d68872"
server_key_iv: "fafc588ae0c64958c5ea625d238b1b01"
server_key_version: "encryptionkey@0"
I20260806 15:54:33.653973 25989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:33.673414 25989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:33.677065 25989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:33.677289 25989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:33.685496 25989 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34829
I20260806 15:54:33.685560 26102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34829 every 8 connection(s)
I20260806 15:54:33.686499 25989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:33.689221 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 25989
I20260806 15:54:33.689388 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:33.689711 26103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:33.689734 14204 external_mini_cluster.cc:1428] Setting key 868412c007b3a2d9a3ec788968fca258
I20260806 15:54:33.696362 26103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Bootstrap starting.
I20260806 15:54:33.699389 26103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Neither blocks nor log segments found. Creating new log.
Aug 06 15:54:33 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:33.700207 26103 log.cc:824] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:33.702354 26103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: No bootstrap required, opened a new log
I20260806 15:54:33.706446 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:33.691260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.13.223.62:34829)
0806 15:54:33.691730 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:33.691750 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:33.691787 (+ 37us) server_negotiation.cc:407] Connection header received
0806 15:54:33.692755 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:33.692786 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:33.693167 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:33.693546 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:33.694520 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.695606 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.696671 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.697188 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.697319 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:33.700437 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:33.700444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:33.700478 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:33.700543 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:33.702987 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:33.703513 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:33.703519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:33.703524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:33.703708 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:33.704006 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:33.704008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:33.704009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:33.704410 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:33.704682 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:33.704952 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 15:54:33.705307 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260806 15:54:33.706442 26103 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32a946d5a3a4a74a93d2c47501415b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34829 } }
I20260806 15:54:33.706789 26103 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:33.706856 26103 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32a946d5a3a4a74a93d2c47501415b0, State: Initialized, Role: FOLLOWER
I20260806 15:54:33.707350 26103 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32a946d5a3a4a74a93d2c47501415b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34829 } }
I20260806 15:54:33.707557 26103 raft_consensus.cc:388] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:33.707636 26103 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:33.707741 26103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:33.708927 26103 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32a946d5a3a4a74a93d2c47501415b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34829 } }
I20260806 15:54:33.709301 26103 leader_election.cc:302] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32a946d5a3a4a74a93d2c47501415b0; no voters:
I20260806 15:54:33.709661 26103 leader_election.cc:288] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:33.709735 26112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:33.709973 26112 raft_consensus.cc:686] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [term 1 LEADER]: Becoming Leader. State: Replica: e32a946d5a3a4a74a93d2c47501415b0, State: Running, Role: LEADER
I20260806 15:54:33.710289 26112 consensus_queue.cc:237] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32a946d5a3a4a74a93d2c47501415b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34829 } }
I20260806 15:54:33.711143 26103 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:33.712389 26113 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e32a946d5a3a4a74a93d2c47501415b0. Latest consensus state: current_term: 1 leader_uuid: "e32a946d5a3a4a74a93d2c47501415b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32a946d5a3a4a74a93d2c47501415b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34829 } } }
I20260806 15:54:33.712569 26113 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:33.712666 26115 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32a946d5a3a4a74a93d2c47501415b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32a946d5a3a4a74a93d2c47501415b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34829 } } }
I20260806 15:54:33.712792 26115 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0 [sys.catalog]: This master's current role is: LEADER
W20260806 15:54:33.716800 26137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:54:33.716892 26137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:54:33.716975 26135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:33.717702 26135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:33.722831 26135 catalog_manager.cc:1347] Generated new cluster ID: f6e22b1bc1b647279621fe23f560b01e
I20260806 15:54:33.722903 26135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:33.743279 26135 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:33.744342 26135 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:33.750450 26135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Generated new TSK 0
I20260806 15:54:33.751176 26135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab.
Entry for principal oryx/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab.
I20260806 15:54:33.783751 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34829
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:33.899957 26149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:33.900224 26149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:33.900270 26149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:33.904359 26149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:33.904431 26149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:33.904472 26149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:33.904572 26149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:33.909866 26149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:33.910964 26149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:33.912053 26149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:33.922195 26159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:33.922184 26158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:33.922392 26149 server_base.cc:1034] running on GCE node
W20260806 15:54:33.922184 26161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:33.922883 26149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:33.923624 26149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:33.924803 26149 hybrid_clock.cc:648] HybridClock initialized: now 1786031673924753 us; error 69 us; skew 500 ppm
Aug 06 15:54:33 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031673, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:33.927740 26149 init.cc:377] Logged in from keytab as oryx/127.13.223.1@KRBTEST.COM (short username oryx)
I20260806 15:54:33.928798 26149 webserver.cc:460] Webserver started at http://127.13.223.1:39385/ using document root <none> and password file <none>
I20260806 15:54:33.929378 26149 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:33.929472 26149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:33.929719 26149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:33.931859 26149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "9a816b1c765548bf9594026a42f90530"
format_stamp: "Formatted at 2026-08-06 15:54:33 on dist-test-slave-rwrg"
server_key: "cd40943af449e13255ab331ced24b376"
server_key_iv: "c376dea828cb418b8c71644f9b6786b4"
server_key_version: "encryptionkey@0"
I20260806 15:54:33.932443 26149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "9a816b1c765548bf9594026a42f90530"
format_stamp: "Formatted at 2026-08-06 15:54:33 on dist-test-slave-rwrg"
server_key: "cd40943af449e13255ab331ced24b376"
server_key_iv: "c376dea828cb418b8c71644f9b6786b4"
server_key_version: "encryptionkey@0"
I20260806 15:54:33.936434 26149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 15:54:33.939071 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:33.940426 26149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:33.940649 26149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "9a816b1c765548bf9594026a42f90530"
format_stamp: "Formatted at 2026-08-06 15:54:33 on dist-test-slave-rwrg"
server_key: "cd40943af449e13255ab331ced24b376"
server_key_iv: "c376dea828cb418b8c71644f9b6786b4"
server_key_version: "encryptionkey@0"
I20260806 15:54:33.940755 26149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:33.963074 26149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:33.966399 26149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:33.966655 26149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:33.968314 26149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:33.968397 26149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:33.968480 26149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:33.968562 26149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 06 15:54:33 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031673, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:33.981348 26149 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:45947
I20260806 15:54:33.981451 26348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:45947 every 8 connection(s)
I20260806 15:54:33.982527 26149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:33.983927 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:33.969224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:59413 (local address 127.13.223.62:34829)
0806 15:54:33.969360 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:33.969364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:33.970661 (+ 1297us) server_negotiation.cc:407] Connection header received
0806 15:54:33.971776 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:33.971779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:33.971838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:33.971904 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:33.973527 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.974393 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.975663 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.975915 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.976028 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:33.979812 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:33.979815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:33.979817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:33.979842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:33.981590 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:33.982241 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:33.982244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:33.982245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:33.982326 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:33.982748 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:33.982751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:33.982753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:33.982979 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:33.983113 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:33.983685 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 15:54:33.983803 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 15:54:33.984886 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:33.969851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.13.223.1:59413)
0806 15:54:33.970396 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:33.970443 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:33.971504 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:33.972386 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:33.972393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:33.972858 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:33.973369 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:33.973383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.974553 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.974557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:33.975533 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:33.975543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.976098 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.976103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:33.976293 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:33.976992 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:33.977030 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:33.979630 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:33.981733 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:33.981739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:33.981753 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:33.982125 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:33.982428 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:33.982433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:33.982435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:33.982625 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:33.983168 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:33.983174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:33.983461 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 15:54:33.983623 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 15:54:33.983873 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260806 15:54:33.986202 26350 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34829
I20260806 15:54:33.986344 26350 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:33.986892 26350 heartbeater.cc:511] Master 127.13.223.62:34829 requested a full tablet report, sending...
I20260806 15:54:33.988569 26028 ts_manager.cc:194] Registered new tserver with Master: 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947)
I20260806 15:54:33.990005 26028 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:59413
I20260806 15:54:33.991534 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 26149
I20260806 15:54:33.991747 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:33.992096 14204 external_mini_cluster.cc:1428] Setting key e76abe10de63cb187f811936c70e995c
I20260806 15:54:33.996104 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:33.988640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:50705 (local address 127.13.223.62:34829)
0806 15:54:33.988764 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:33.988768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:33.988973 (+ 205us) server_negotiation.cc:407] Connection header received
0806 15:54:33.989058 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:33.989060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:33.989099 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:33.989166 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:33.989717 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.990291 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.991279 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.991583 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.991687 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:33.993060 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:33.993063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:33.993064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:33.993085 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:33.994437 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:33.994929 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:33.994932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:33.994933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:33.995005 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:33.995321 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:33.995323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:33.995324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:33.995472 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:33.995551 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:33.995864 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 15:54:33.995949 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:54:33.996268 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:33.988543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.13.223.1:50705)
0806 15:54:33.988867 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:33.988882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:33.988978 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:33.989207 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:33.989210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:33.989459 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:33.989589 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:33.989596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.990415 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.990419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:33.991158 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:33.991167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:33.991810 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:33.991817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:33.991914 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:33.992463 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:33.992478 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:33.992923 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:33.994549 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:33.994553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:33.994555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:33.994811 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:33.995113 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:33.995119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:33.995124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:33.995228 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:33.995567 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:33.995573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:33.995697 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 15:54:33.995817 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 15:54:33.995993 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab.
Entry for principal oryx/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab.
I20260806 15:54:34.021786 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34829
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:34.134476 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:34.134781 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:34.134871 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:34.138867 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:34.138960 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:34.139032 26360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:34.139111 26360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:34.144345 26360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:34.145471 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:34.146476 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:34.156747 26367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:34.156730 26369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:34.156747 26365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:34.156977 26360 server_base.cc:1034] running on GCE node
I20260806 15:54:34.157532 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:34.158241 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:34.159400 26360 hybrid_clock.cc:648] HybridClock initialized: now 1786031674159379 us; error 30 us; skew 500 ppm
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031674, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:34.162184 26360 init.cc:377] Logged in from keytab as oryx/127.13.223.2@KRBTEST.COM (short username oryx)
I20260806 15:54:34.163061 26360 webserver.cc:460] Webserver started at http://127.13.223.2:38141/ using document root <none> and password file <none>
I20260806 15:54:34.163615 26360 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:34.163707 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:34.163997 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:34.166133 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "43eb9cc094014655b16fe4d6df8083a0"
format_stamp: "Formatted at 2026-08-06 15:54:34 on dist-test-slave-rwrg"
server_key: "17184ab4befe310c5fc9275c522f44b0"
server_key_iv: "2df0ff70294494062b914080457e324d"
server_key_version: "encryptionkey@0"
I20260806 15:54:34.166688 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "43eb9cc094014655b16fe4d6df8083a0"
format_stamp: "Formatted at 2026-08-06 15:54:34 on dist-test-slave-rwrg"
server_key: "17184ab4befe310c5fc9275c522f44b0"
server_key_iv: "2df0ff70294494062b914080457e324d"
server_key_version: "encryptionkey@0"
I20260806 15:54:34.170557 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:34.172931 26380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:34.174120 26360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:34.174259 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "43eb9cc094014655b16fe4d6df8083a0"
format_stamp: "Formatted at 2026-08-06 15:54:34 on dist-test-slave-rwrg"
server_key: "17184ab4befe310c5fc9275c522f44b0"
server_key_iv: "2df0ff70294494062b914080457e324d"
server_key_version: "encryptionkey@0"
I20260806 15:54:34.174371 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:34.184967 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:34.187942 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:34.188150 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:34.189603 26360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:34.189652 26360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:34.189716 26360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:34.189749 26360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031674, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:34.200790 26360 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:39201
I20260806 15:54:34.200852 26557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:39201 every 8 connection(s)
I20260806 15:54:34.201922 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:34.204437 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.190399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:58177 (local address 127.13.223.62:34829)
0806 15:54:34.190561 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.190566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.191424 (+ 858us) server_negotiation.cc:407] Connection header received
0806 15:54:34.192581 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.192586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.192657 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.192765 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:34.194192 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.195141 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.197090 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.197395 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.197483 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.200775 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:34.200779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:34.200782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:34.200811 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:34.202356 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.202982 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.202985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.202986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.203079 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.203383 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.203385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.203386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.203591 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:34.203671 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.204198 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.204317 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 15:54:34.205227 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.190798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.13.223.2:58177)
0806 15:54:34.191250 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.191299 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.192300 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.192934 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.192943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.193356 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:34.194052 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.194076 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.195293 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.195297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.196944 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.196956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.197576 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.197582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.197814 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.198580 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:34.198604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:34.200589 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:34.202514 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.202520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.202534 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.202877 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.203168 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.203170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.203171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.203299 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.203681 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:34.203686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:34.203901 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.204068 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.204320 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260806 15:54:34.206694 26558 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34829
I20260806 15:54:34.206907 26558 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:34.207461 26558 heartbeater.cc:511] Master 127.13.223.62:34829 requested a full tablet report, sending...
I20260806 15:54:34.208789 26029 ts_manager.cc:194] Registered new tserver with Master: 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:34.208936 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 26360
I20260806 15:54:34.209065 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:34.209475 14204 external_mini_cluster.cc:1428] Setting key 3d32609e94d41b2675e30d7678056e9a
I20260806 15:54:34.209724 26029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:58177
I20260806 15:54:34.217572 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.208841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:47389 (local address 127.13.223.62:34829)
0806 15:54:34.208958 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.208962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.209189 (+ 227us) server_negotiation.cc:407] Connection header received
0806 15:54:34.209354 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.209357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.209406 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.209484 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:34.210111 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.210817 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.212035 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.212264 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.212376 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.213895 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:34.213898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:34.213900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:34.213925 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:34.215563 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.216109 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.216112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.216114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.216207 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.216601 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.216607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.216612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.216858 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:34.216958 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.217290 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.217397 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:54:34.217706 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.208791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.13.223.2:47389)
0806 15:54:34.209108 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.209120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.209226 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.209502 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.209505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.209711 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:34.209802 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.209808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.210974 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.210977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.211894 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.211904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.212564 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.212568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.212657 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.213244 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:34.213256 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:34.213754 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:34.215696 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.215702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.215707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.215978 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.216320 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.216326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.216331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.216441 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.216974 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:34.216978 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:34.217084 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.217225 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.217393 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab.
Entry for principal oryx/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab.
I20260806 15:54:34.243260 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34829
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:34.358604 26569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:34.358911 26569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:34.359006 26569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:34.363126 26569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:34.363225 26569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:34.363314 26569 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 15:54:34.363400 26569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:34.368949 26569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:34.370105 26569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:34.371172 26569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:34.381490 26581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:34.381517 26569 server_base.cc:1034] running on GCE node
W20260806 15:54:34.381465 26576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:34.381451 26577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:34.382186 26569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:34.382886 26569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:34.384093 26569 hybrid_clock.cc:648] HybridClock initialized: now 1786031674384077 us; error 32 us; skew 500 ppm
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031674, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:34.387215 26569 init.cc:377] Logged in from keytab as oryx/127.13.223.3@KRBTEST.COM (short username oryx)
I20260806 15:54:34.388139 26569 webserver.cc:460] Webserver started at http://127.13.223.3:39905/ using document root <none> and password file <none>
I20260806 15:54:34.388782 26569 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:34.388875 26569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:34.389118 26569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:34.391196 26569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "b071c370dbf141be97fe0948ffeca35d"
format_stamp: "Formatted at 2026-08-06 15:54:34 on dist-test-slave-rwrg"
server_key: "570de4a5f5908eff473807544c3359c7"
server_key_iv: "141ddaa22933cad9c4c81f8633c84f0a"
server_key_version: "encryptionkey@0"
I20260806 15:54:34.391778 26569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "b071c370dbf141be97fe0948ffeca35d"
format_stamp: "Formatted at 2026-08-06 15:54:34 on dist-test-slave-rwrg"
server_key: "570de4a5f5908eff473807544c3359c7"
server_key_iv: "141ddaa22933cad9c4c81f8633c84f0a"
server_key_version: "encryptionkey@0"
I20260806 15:54:34.395546 26569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 15:54:34.398240 26591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:34.399596 26569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:54:34.399770 26569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "b071c370dbf141be97fe0948ffeca35d"
format_stamp: "Formatted at 2026-08-06 15:54:34 on dist-test-slave-rwrg"
server_key: "570de4a5f5908eff473807544c3359c7"
server_key_iv: "141ddaa22933cad9c4c81f8633c84f0a"
server_key_version: "encryptionkey@0"
I20260806 15:54:34.399909 26569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:34.429754 26569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:34.433167 26569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:34.433378 26569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:34.434855 26569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:34.434930 26569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:34.435007 26569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:34.435055 26569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031674, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:34.446688 26569 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40751
I20260806 15:54:34.446732 26763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40751 every 8 connection(s)
I20260806 15:54:34.447715 26569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:34.449527 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.435885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:44771 (local address 127.13.223.62:34829)
0806 15:54:34.436017 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.436020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.437177 (+ 1157us) server_negotiation.cc:407] Connection header received
0806 15:54:34.438300 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.438307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.438377 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.438479 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:34.439699 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.440628 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.441827 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.442061 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.442176 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.445705 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:34.445710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:34.445715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:34.445745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:34.447321 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.447994 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.447997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.447998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.448111 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.448529 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.448532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.448534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.448772 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:34.448854 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.449308 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.449408 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:54:34.450315 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.436454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.13.223.3:44771)
0806 15:54:34.437032 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.437075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.438037 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.438660 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.438671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.439084 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:34.439550 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.439563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.440748 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.440753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.441692 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.441702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.442229 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.442233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.442398 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.442971 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:34.442986 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:34.445525 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:34.447439 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.447445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.447459 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.447903 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.448236 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.448239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.448241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.448397 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.448869 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:34.448874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:34.449090 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.449244 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.449498 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":142,"threads_started":1}
I20260806 15:54:34.450861 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 26569
I20260806 15:54:34.451010 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:34.451335 14204 external_mini_cluster.cc:1428] Setting key 7d27ce8fdfbaa4d56d122d7e661973ed
I20260806 15:54:34.451628 26765 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34829
I20260806 15:54:34.451768 26765 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:34.452360 26765 heartbeater.cc:511] Master 127.13.223.62:34829 requested a full tablet report, sending...
I20260806 15:54:34.453490 26029 ts_manager.cc:194] Registered new tserver with Master: b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:34.454241 26029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:44771
I20260806 15:54:34.461395 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.453968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:37371 (local address 127.13.223.62:34829)
0806 15:54:34.454084 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.454088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.454719 (+ 631us) server_negotiation.cc:407] Connection header received
0806 15:54:34.454849 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.454852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.454898 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.454980 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:34.455373 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.456090 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.456968 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.457188 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.457262 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.458316 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:34.458318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:34.458320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:34.458341 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:34.459760 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.460238 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.460240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.460242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.460315 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.460652 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.460655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.460656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.460814 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:34.460888 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.461204 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.461288 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 15:54:34.461414 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.453905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.13.223.3:37371)
0806 15:54:34.454629 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.454644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.454744 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.454988 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.454990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.455166 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:34.455262 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.455266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.456217 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.456220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.456867 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.456877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.457268 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.457270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.457367 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.457861 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:34.457872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:34.458166 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:34.459903 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.459907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.459908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.460140 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.460404 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:34.460406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:34.460408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:34.460552 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:34.460901 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:34.460904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:34.460990 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.461152 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.461274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":90,"threads_started":1}
I20260806 15:54:34.465281 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 15:54:34.491544 26108 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:34.485070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.13.223.62:34829)
0806 15:54:34.485207 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.485210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.485224 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:34.485268 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.485270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.485309 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.485385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:34.486313 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.486872 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.487880 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.488090 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.488177 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.491364 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44680: BlockingRecv error: recv got EOF from 127.0.0.1:44680 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 15:54:34 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:34.501315 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.493043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.13.223.62:34829)
0806 15:54:34.493182 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.493186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.493310 (+ 124us) server_negotiation.cc:407] Connection header received
0806 15:54:34.493462 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.493466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.493516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.493604 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:34.494195 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.494698 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.495482 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.495712 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.495822 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.498154 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:34.498158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:34.498160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:34.498187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:34.499602 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.500048 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.500052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.500053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.500134 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:34.500412 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:34.500416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:34.500418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:34.500654 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:34.500797 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.501027 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.501154 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:54:34.504148 26029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:34.507368 26029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:34.522753 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.516288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:39201 (local address 127.0.0.1:40316)
0806 15:54:34.517810 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.517824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.517908 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.518386 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.518389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.518407 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:34.518525 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.518542 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.519608 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.519614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.521287 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.521299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.522289 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.522293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.522389 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.522418 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.522490 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.522562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":812,"thread_start_us":95,"threads_started":1}
I20260806 15:54:34.522912 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.517125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.13.223.3:40751)
0806 15:54:34.517464 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.517478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.517752 (+ 274us) server_negotiation.cc:407] Connection header received
0806 15:54:34.517837 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.517844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.518019 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.518182 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:34.518446 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.519580 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.521193 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.521926 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.522038 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.522204 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.522648 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.522759 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":141,"threads_started":1}
I20260806 15:54:34.522907 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.516288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45947 (local address 127.0.0.1:33354)
0806 15:54:34.517014 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.517066 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.517218 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.518019 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.518024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.518044 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:34.518178 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.518184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.519682 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.519685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.521470 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.521478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.522511 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.522515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.522603 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.522627 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.522713 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.522771 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":403,"thread_start_us":92,"threads_started":1}
I20260806 15:54:34.523111 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.516584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40751 (local address 127.0.0.1:42428)
0806 15:54:34.517582 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.517600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.517713 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.518179 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.518182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.518208 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:34.518315 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.518322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.519663 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.519666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.521058 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.521068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.522064 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.522068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.522187 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.522241 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.522354 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.522502 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":160,"thread_start_us":102,"threads_started":1}
I20260806 15:54:34.523542 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.516547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.13.223.2:39201)
0806 15:54:34.517975 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.517991 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.518012 (+ 21us) server_negotiation.cc:407] Connection header received
0806 15:54:34.518085 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.518092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.518252 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.518393 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:34.518654 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.519496 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.521428 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.522084 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.522188 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.522354 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.523317 (+ 963us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.523410 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":94,"threads_started":1}
I20260806 15:54:34.523553 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.516432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.13.223.1:45947)
0806 15:54:34.517069 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.517076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.517158 (+ 82us) server_negotiation.cc:407] Connection header received
0806 15:54:34.517429 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.517436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.517648 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.517829 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:34.518312 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.519533 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.521602 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.522386 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.522504 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.522631 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.522741 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.523432 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":160,"threads_started":1}
I20260806 15:54:34.525944 26670 tablet_service.cc:1467] Processing CreateTablet for tablet ef330e4ffed943e1b8784821f1e830ea (DEFAULT_TABLE table=test-table [id=53960538383e4143aa66ee36d4d75ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:34.526268 26461 tablet_service.cc:1467] Processing CreateTablet for tablet ef330e4ffed943e1b8784821f1e830ea (DEFAULT_TABLE table=test-table [id=53960538383e4143aa66ee36d4d75ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:34.526265 26244 tablet_service.cc:1467] Processing CreateTablet for tablet ef330e4ffed943e1b8784821f1e830ea (DEFAULT_TABLE table=test-table [id=53960538383e4143aa66ee36d4d75ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:34.527064 26670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef330e4ffed943e1b8784821f1e830ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:34.527271 26244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef330e4ffed943e1b8784821f1e830ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:34.527271 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef330e4ffed943e1b8784821f1e830ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:34.533425 26810 tablet_bootstrap.cc:492] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Bootstrap starting.
I20260806 15:54:34.535108 26812 tablet_bootstrap.cc:492] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Bootstrap starting.
I20260806 15:54:34.535467 26813 tablet_bootstrap.cc:492] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Bootstrap starting.
I20260806 15:54:34.535856 26810 tablet_bootstrap.cc:654] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:34.536672 26810 log.cc:824] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:34.537245 26812 tablet_bootstrap.cc:654] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:34.537323 26813 tablet_bootstrap.cc:654] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:34.538120 26813 log.cc:824] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:34.538126 26812 log.cc:824] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:34.538602 26810 tablet_bootstrap.cc:492] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: No bootstrap required, opened a new log
I20260806 15:54:34.538826 26810 ts_tablet_manager.cc:1397] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260806 15:54:34.539990 26812 tablet_bootstrap.cc:492] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: No bootstrap required, opened a new log
I20260806 15:54:34.540215 26812 ts_tablet_manager.cc:1397] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 15:54:34.540752 26813 tablet_bootstrap.cc:492] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: No bootstrap required, opened a new log
I20260806 15:54:34.540936 26813 ts_tablet_manager.cc:1397] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260806 15:54:34.542878 26810 raft_consensus.cc:348] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.543149 26810 raft_consensus.cc:374] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:34.543215 26810 raft_consensus.cc:729] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Initialized, Role: FOLLOWER
I20260806 15:54:34.543678 26810 consensus_queue.cc:260] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.543980 26812 raft_consensus.cc:348] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.544268 26812 raft_consensus.cc:374] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:34.544341 26812 raft_consensus.cc:729] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a816b1c765548bf9594026a42f90530, State: Initialized, Role: FOLLOWER
I20260806 15:54:34.544536 26765 heartbeater.cc:503] Master 127.13.223.62:34829 was elected leader, sending a full tablet report...
I20260806 15:54:34.544668 26813 raft_consensus.cc:348] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.544862 26810 ts_tablet_manager.cc:1428] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260806 15:54:34.544910 26813 raft_consensus.cc:374] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:34.544965 26813 raft_consensus.cc:729] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eb9cc094014655b16fe4d6df8083a0, State: Initialized, Role: FOLLOWER
I20260806 15:54:34.544919 26812 consensus_queue.cc:260] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.545846 26812 ts_tablet_manager.cc:1428] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 15:54:34.546198 26350 heartbeater.cc:503] Master 127.13.223.62:34829 was elected leader, sending a full tablet report...
I20260806 15:54:34.545801 26813 consensus_queue.cc:260] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.547852 26558 heartbeater.cc:503] Master 127.13.223.62:34829 was elected leader, sending a full tablet report...
I20260806 15:54:34.548342 26813 ts_tablet_manager.cc:1428] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260806 15:54:34.602054 26818 raft_consensus.cc:482] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:34.602301 26818 raft_consensus.cc:504] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.603492 26818 leader_election.cc:288] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201), b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:34.607717 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.603912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:33013 (local address 127.13.223.2:39201)
0806 15:54:34.604078 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.604083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.604225 (+ 142us) server_negotiation.cc:407] Connection header received
0806 15:54:34.604392 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.604403 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.604446 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.604592 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:34.604904 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.605448 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.606696 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.607082 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.607176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.607256 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.607521 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.607591 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 15:54:34.607730 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.603915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:37587 (local address 127.13.223.3:40751)
0806 15:54:34.604182 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.604186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.604343 (+ 157us) server_negotiation.cc:407] Connection header received
0806 15:54:34.604533 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.604538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.604601 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.604678 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:34.604979 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.605563 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.606726 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.607119 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.607258 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.607313 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.607525 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.607607 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260806 15:54:34.607755 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.603770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40751 (local address 127.13.223.1:37587)
0806 15:54:34.604238 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.604253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.604385 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.604717 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.604719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.604744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:34.604864 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.604870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.605687 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.605690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.606579 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.606589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.607278 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.607282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.607378 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.607428 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.607508 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.607579 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":62,"thread_start_us":135,"threads_started":1}
I20260806 15:54:34.607766 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.603770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:39201 (local address 127.13.223.1:33013)
0806 15:54:34.604117 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:34.604133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:34.604273 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:34.604622 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:34.604627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:34.604648 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:34.604785 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.604797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.605557 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.605560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.606573 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:34.606583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.607211 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.607218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:34.607308 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.607342 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 15:54:34.607503 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 15:54:34.607597 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":29,"thread_start_us":126,"threads_started":1}
I20260806 15:54:34.608461 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef330e4ffed943e1b8784821f1e830ea" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0" is_pre_election: true
I20260806 15:54:34.608605 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef330e4ffed943e1b8784821f1e830ea" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b071c370dbf141be97fe0948ffeca35d" is_pre_election: true
I20260806 15:54:34.608851 26491 raft_consensus.cc:2454] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a816b1c765548bf9594026a42f90530 in term 0.
I20260806 15:54:34.608914 26701 raft_consensus.cc:2454] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a816b1c765548bf9594026a42f90530 in term 0.
I20260806 15:54:34.609364 26176 leader_election.cc:302] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, 9a816b1c765548bf9594026a42f90530, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:34.609628 26818 raft_consensus.cc:2781] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:34.609720 26818 raft_consensus.cc:482] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:34.609773 26818 raft_consensus.cc:3037] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:34.610817 26818 raft_consensus.cc:504] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.611164 26818 leader_election.cc:288] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 election: Requested vote from peers 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201), b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:34.611503 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef330e4ffed943e1b8784821f1e830ea" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0"
I20260806 15:54:34.611548 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef330e4ffed943e1b8784821f1e830ea" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b071c370dbf141be97fe0948ffeca35d"
I20260806 15:54:34.611653 26491 raft_consensus.cc:3037] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:34.611668 26701 raft_consensus.cc:3037] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:34.612977 26491 raft_consensus.cc:2454] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a816b1c765548bf9594026a42f90530 in term 1.
I20260806 15:54:34.613312 26176 leader_election.cc:302] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, 9a816b1c765548bf9594026a42f90530; no voters:
I20260806 15:54:34.613356 26701 raft_consensus.cc:2454] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a816b1c765548bf9594026a42f90530 in term 1.
I20260806 15:54:34.613518 26818 raft_consensus.cc:2781] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:34.613785 26818 raft_consensus.cc:686] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 1 LEADER]: Becoming Leader. State: Replica: 9a816b1c765548bf9594026a42f90530, State: Running, Role: LEADER
I20260806 15:54:34.614143 26818 consensus_queue.cc:237] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.617571 26026 catalog_manager.cc:5495] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a816b1c765548bf9594026a42f90530 (127.13.223.1). New cstate: current_term: 1 leader_uuid: "9a816b1c765548bf9594026a42f90530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:34.651435 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:34.648043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.13.223.1:45947)
0806 15:54:34.648192 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:34.648195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:34.648209 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:34.648336 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:34.648338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:34.648395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:34.648482 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:34.648776 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.649275 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.650055 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:34.650259 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:34.650336 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:34.650678 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:34.651103 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:34.651189 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:34.651279 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 15:54:34.651328 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 15:54:34.657436 26701 raft_consensus.cc:1261] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Refusing update from remote peer 9a816b1c765548bf9594026a42f90530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:34.657436 26491 raft_consensus.cc:1261] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Refusing update from remote peer 9a816b1c765548bf9594026a42f90530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:34.658156 26818 consensus_queue.cc:1035] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:34.658399 26823 consensus_queue.cc:1035] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:34.669155 26829 mvcc.cc:205] Tried to move back new op lower bound from 7315585739389431808 to 7315585739222167552. Current Snapshot: MvccSnapshot[applied={T|T < 7315585739389431808}]
I20260806 15:54:34.669514 26828 mvcc.cc:205] Tried to move back new op lower bound from 7315585739389431808 to 7315585739222167552. Current Snapshot: MvccSnapshot[applied={T|T < 7315585739389431808}]
I20260806 15:54:34.675549 26026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:34.675717 26026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:34.687230 26026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Sending DeleteTablet for 3 replicas of tablet ef330e4ffed943e1b8784821f1e830ea
I20260806 15:54:34.688110 26244 tablet_service.cc:1514] Processing DeleteTablet for tablet ef330e4ffed943e1b8784821f1e830ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:34 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33354
I20260806 15:54:34.688179 26670 tablet_service.cc:1514] Processing DeleteTablet for tablet ef330e4ffed943e1b8784821f1e830ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:34 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:42428
I20260806 15:54:34.688985 26461 tablet_service.cc:1514] Processing DeleteTablet for tablet ef330e4ffed943e1b8784821f1e830ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:34 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40316
I20260806 15:54:34.689723 26026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:34.690157 26026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:34.690866 26842 tablet_replica.cc:330] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: stopping tablet replica
I20260806 15:54:34.691201 26842 raft_consensus.cc:2229] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:34.691596 26842 raft_consensus.cc:2258] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:34.693933 26842 ts_tablet_manager.cc:1905] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:34.695204 26244 tablet_service.cc:1467] Processing CreateTablet for tablet 24eb99d9ad004d4cb06cfa9cfed42264 (DEFAULT_TABLE table=test-table [id=0b77470eef3347a89dcaa1314103aab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:34.695461 26244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb99d9ad004d4cb06cfa9cfed42264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:34.695571 26670 tablet_service.cc:1467] Processing CreateTablet for tablet 24eb99d9ad004d4cb06cfa9cfed42264 (DEFAULT_TABLE table=test-table [id=0b77470eef3347a89dcaa1314103aab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:34.695838 26670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb99d9ad004d4cb06cfa9cfed42264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:34.697245 26461 tablet_service.cc:1467] Processing CreateTablet for tablet 24eb99d9ad004d4cb06cfa9cfed42264 (DEFAULT_TABLE table=test-table [id=0b77470eef3347a89dcaa1314103aab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:34.697511 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb99d9ad004d4cb06cfa9cfed42264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:34.698125 26812 tablet_bootstrap.cc:492] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Bootstrap starting.
I20260806 15:54:34.699106 26812 tablet_bootstrap.cc:654] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:34.699819 26813 tablet_bootstrap.cc:492] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Bootstrap starting.
I20260806 15:54:34.700819 26813 tablet_bootstrap.cc:654] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:34.701210 26812 tablet_bootstrap.cc:492] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: No bootstrap required, opened a new log
I20260806 15:54:34.701280 26812 ts_tablet_manager.cc:1397] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:34.701550 26843 tablet_replica.cc:330] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: stopping tablet replica
I20260806 15:54:34.701737 26812 raft_consensus.cc:348] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.701851 26812 raft_consensus.cc:374] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:34.701853 26843 raft_consensus.cc:2229] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:34.701882 26812 raft_consensus.cc:729] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a816b1c765548bf9594026a42f90530, State: Initialized, Role: FOLLOWER
I20260806 15:54:34.701999 26812 consensus_queue.cc:260] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.702224 26812 ts_tablet_manager.cc:1428] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:34.702265 26810 tablet_bootstrap.cc:492] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Bootstrap starting.
I20260806 15:54:34.705046 26810 tablet_bootstrap.cc:654] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:34.702167 26843 raft_consensus.cc:2258] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:34.706746 26810 tablet_bootstrap.cc:492] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: No bootstrap required, opened a new log
I20260806 15:54:34.706848 26810 ts_tablet_manager.cc:1397] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 15:54:34.707243 26842 ts_tablet_manager.cc:1918] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:34.707355 26842 log.cc:1196] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/ef330e4ffed943e1b8784821f1e830ea
I20260806 15:54:34.707406 26810 raft_consensus.cc:348] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.707545 26810 raft_consensus.cc:374] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:34.707605 26810 raft_consensus.cc:729] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Initialized, Role: FOLLOWER
I20260806 15:54:34.707788 26842 ts_tablet_manager.cc:1939] T ef330e4ffed943e1b8784821f1e830ea P 9a816b1c765548bf9594026a42f90530: Deleting consensus metadata
I20260806 15:54:34.707770 26810 consensus_queue.cc:260] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.708055 26810 ts_tablet_manager.cc:1428] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 15:54:34.709367 26813 tablet_bootstrap.cc:492] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: No bootstrap required, opened a new log
I20260806 15:54:34.709467 26813 ts_tablet_manager.cc:1397] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260806 15:54:34.709959 26813 raft_consensus.cc:348] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.710083 26813 raft_consensus.cc:374] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:34.710125 26813 raft_consensus.cc:729] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eb9cc094014655b16fe4d6df8083a0, State: Initialized, Role: FOLLOWER
I20260806 15:54:34.710269 26813 consensus_queue.cc:260] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:34.710546 26813 ts_tablet_manager.cc:1428] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:34.711694 26009 catalog_manager.cc:4841] TS 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947): tablet ef330e4ffed943e1b8784821f1e830ea (table test-table [id=53960538383e4143aa66ee36d4d75ff8]) successfully deleted
I20260806 15:54:34.712240 26843 ts_tablet_manager.cc:1905] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:34.713387 26844 tablet_replica.cc:330] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: stopping tablet replica
I20260806 15:54:34.713771 26844 raft_consensus.cc:2229] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:34.714136 26844 raft_consensus.cc:2258] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:34.715009 26843 ts_tablet_manager.cc:1918] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:34.715107 26843 log.cc:1196] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/ef330e4ffed943e1b8784821f1e830ea
I20260806 15:54:34.715417 26843 ts_tablet_manager.cc:1939] T ef330e4ffed943e1b8784821f1e830ea P b071c370dbf141be97fe0948ffeca35d: Deleting consensus metadata
I20260806 15:54:34.716660 26007 catalog_manager.cc:4841] TS b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751): tablet ef330e4ffed943e1b8784821f1e830ea (table test-table [id=53960538383e4143aa66ee36d4d75ff8]) successfully deleted
I20260806 15:54:34.717424 26844 ts_tablet_manager.cc:1905] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:34.719704 26844 ts_tablet_manager.cc:1918] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:34.719785 26844 log.cc:1196] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/ef330e4ffed943e1b8784821f1e830ea
I20260806 15:54:34.720144 26844 ts_tablet_manager.cc:1939] T ef330e4ffed943e1b8784821f1e830ea P 43eb9cc094014655b16fe4d6df8083a0: Deleting consensus metadata
I20260806 15:54:34.721356 26010 catalog_manager.cc:4841] TS 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201): tablet ef330e4ffed943e1b8784821f1e830ea (table test-table [id=53960538383e4143aa66ee36d4d75ff8]) successfully deleted
W20260806 15:54:34.734040 26351 tablet.cc:2369] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:34.949499 26766 tablet.cc:2369] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:54:34.953908 26559 tablet.cc:2369] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:54:35.064666 26817 raft_consensus.cc:482] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:35.064841 26817 raft_consensus.cc:504] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.065979 26817 leader_election.cc:288] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:35.070007 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.066377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:37331 (local address 127.13.223.2:39201)
0806 15:54:35.066509 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.066514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.066826 (+ 312us) server_negotiation.cc:407] Connection header received
0806 15:54:35.066956 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.066959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.067003 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.067081 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:35.067356 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.067853 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.068974 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.069388 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.069504 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.069556 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.069797 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.069872 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 15:54:35.070026 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.066230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45947 (local address 127.13.223.3:59075)
0806 15:54:35.066540 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.066552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.066650 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.066924 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.066929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.066946 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:35.067084 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.067090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.067851 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.067854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.068847 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.068856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.069546 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.069550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.069690 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.069708 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.069811 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.069878 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":88,"thread_start_us":77,"threads_started":1}
I20260806 15:54:35.070031 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.066341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:59075 (local address 127.13.223.1:45947)
0806 15:54:35.066509 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.066514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.066627 (+ 113us) server_negotiation.cc:407] Connection header received
0806 15:54:35.066753 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.066759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.066823 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.066899 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:35.067195 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.067739 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.068973 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.069395 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.069540 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.069591 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.069851 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.069928 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 15:54:35.070031 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.066243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:39201 (local address 127.13.223.3:37331)
0806 15:54:35.066722 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.066737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.066840 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.067108 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.067110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.067124 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:35.067247 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.067253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.067961 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.067964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.068851 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.068860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.069523 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.069527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.069636 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.069686 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.069746 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.069839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260806 15:54:35.070556 26834 raft_consensus.cc:482] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:35.070571 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0" is_pre_election: true
I20260806 15:54:35.070544 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530" is_pre_election: true
I20260806 15:54:35.070688 26834 raft_consensus.cc:504] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.070727 26491 raft_consensus.cc:2454] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 0.
I20260806 15:54:35.070904 26277 raft_consensus.cc:2454] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 0.
I20260806 15:54:35.071112 26834 leader_election.cc:288] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201), b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:35.071136 26595 leader_election.cc:302] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.071431 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b071c370dbf141be97fe0948ffeca35d" is_pre_election: true
I20260806 15:54:35.071508 26817 raft_consensus.cc:2781] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:35.071527 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0" is_pre_election: true
I20260806 15:54:35.071596 26701 raft_consensus.cc:2454] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a816b1c765548bf9594026a42f90530 in term 0.
I20260806 15:54:35.071635 26491 raft_consensus.cc:2454] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a816b1c765548bf9594026a42f90530 in term 0.
I20260806 15:54:35.071659 26817 raft_consensus.cc:482] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:35.071713 26817 raft_consensus.cc:3037] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.071864 26173 leader_election.cc:302] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a816b1c765548bf9594026a42f90530, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.072059 26834 raft_consensus.cc:2781] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:35.072125 26834 raft_consensus.cc:482] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:35.072162 26834 raft_consensus.cc:3037] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.073092 26817 raft_consensus.cc:504] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.073446 26834 raft_consensus.cc:504] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.073522 26817 leader_election.cc:288] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 election: Requested vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:35.073812 26834 leader_election.cc:288] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 election: Requested vote from peers 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201), b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:35.073853 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0"
I20260806 15:54:35.073860 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530"
I20260806 15:54:35.073956 26491 raft_consensus.cc:3037] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.074023 26277 raft_consensus.cc:2379] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b071c370dbf141be97fe0948ffeca35d in current term 1: Already voted for candidate 9a816b1c765548bf9594026a42f90530 in this term.
I20260806 15:54:35.074092 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0"
I20260806 15:54:35.074402 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb99d9ad004d4cb06cfa9cfed42264" candidate_uuid: "9a816b1c765548bf9594026a42f90530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b071c370dbf141be97fe0948ffeca35d"
I20260806 15:54:35.074541 26701 raft_consensus.cc:2379] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a816b1c765548bf9594026a42f90530 in current term 1: Already voted for candidate b071c370dbf141be97fe0948ffeca35d in this term.
I20260806 15:54:35.074815 26173 leader_election.cc:302] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a816b1c765548bf9594026a42f90530; no voters: 43eb9cc094014655b16fe4d6df8083a0, b071c370dbf141be97fe0948ffeca35d
I20260806 15:54:35.075013 26834 raft_consensus.cc:2726] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 15:54:35.075147 26491 raft_consensus.cc:2454] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 1.
I20260806 15:54:35.075455 26595 leader_election.cc:302] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, b071c370dbf141be97fe0948ffeca35d; no voters: 9a816b1c765548bf9594026a42f90530
I20260806 15:54:35.075675 26817 raft_consensus.cc:2781] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:35.075867 26817 raft_consensus.cc:686] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 1 LEADER]: Becoming Leader. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Running, Role: LEADER
I20260806 15:54:35.076120 26817 consensus_queue.cc:237] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.078408 26026 catalog_manager.cc:5495] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d reported cstate change: term changed from 0 to 1, leader changed from <none> to b071c370dbf141be97fe0948ffeca35d (127.13.223.3). New cstate: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } health_report { overall_health: HEALTHY } } }
Aug 06 15:54:35 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031673, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.62@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:35.178701 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.170142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34829 (local address 127.0.0.1:44700)
0806 15:54:35.170447 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.170469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.170605 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.170874 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.170877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.171311 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:35.171412 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.171418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.172133 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.172136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.172852 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.172861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.173280 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.173282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.173340 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.173947 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:35.173959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:35.175474 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:35.177208 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:35.177220 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:35.177225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:35.177454 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:35.177771 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:35.177775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:35.177779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:35.177878 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:35.178215 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:35.178220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:35.178323 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.178415 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.178569 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260806 15:54:35.178826 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.170279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.13.223.62:34829)
0806 15:54:35.170542 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.170548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.170564 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:35.170694 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.170696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.170736 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.170835 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:35.171520 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.172034 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.172951 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.173186 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.173263 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.175621 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:35.175625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:35.175626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:35.175648 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:35.177108 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:35.177591 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:35.177593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:35.177594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:35.177666 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:35.177944 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:35.177946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:35.177947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:35.178111 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:35.178220 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.178479 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.178647 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
I20260806 15:54:35.182793 26026 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:44700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 15:54:35.183503 26026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:35.188308 26244 tablet_service.cc:1467] Processing CreateTablet for tablet f39a7cb45f594a5785a7b69367a11367 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71fc1debb2304ead89d5ea0c10ab3d0f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:35.188453 26670 tablet_service.cc:1467] Processing CreateTablet for tablet f39a7cb45f594a5785a7b69367a11367 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71fc1debb2304ead89d5ea0c10ab3d0f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:35.188555 26461 tablet_service.cc:1467] Processing CreateTablet for tablet f39a7cb45f594a5785a7b69367a11367 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71fc1debb2304ead89d5ea0c10ab3d0f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 15:54:35.188752 26244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f39a7cb45f594a5785a7b69367a11367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.188825 26670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f39a7cb45f594a5785a7b69367a11367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.188889 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f39a7cb45f594a5785a7b69367a11367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.191262 26810 tablet_bootstrap.cc:492] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: Bootstrap starting.
I20260806 15:54:35.191562 26812 tablet_bootstrap.cc:492] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: Bootstrap starting.
I20260806 15:54:35.192201 26813 tablet_bootstrap.cc:492] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: Bootstrap starting.
I20260806 15:54:35.192307 26810 tablet_bootstrap.cc:654] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.192479 26812 tablet_bootstrap.cc:654] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.193198 26813 tablet_bootstrap.cc:654] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.193670 26810 tablet_bootstrap.cc:492] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: No bootstrap required, opened a new log
I20260806 15:54:35.193779 26810 ts_tablet_manager.cc:1397] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:35.194275 26810 raft_consensus.cc:348] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.194406 26810 raft_consensus.cc:374] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.194440 26813 tablet_bootstrap.cc:492] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: No bootstrap required, opened a new log
I20260806 15:54:35.194448 26810 raft_consensus.cc:729] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.194545 26813 ts_tablet_manager.cc:1397] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:35.194577 26810 consensus_queue.cc:260] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.194870 26810 ts_tablet_manager.cc:1428] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:35.195056 26813 raft_consensus.cc:348] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.195185 26813 raft_consensus.cc:374] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.195230 26813 raft_consensus.cc:729] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eb9cc094014655b16fe4d6df8083a0, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.195209 26817 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.195353 26817 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.195362 26813 consensus_queue.cc:260] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.195681 26819 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.195801 26819 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.196302 26812 tablet_bootstrap.cc:492] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: No bootstrap required, opened a new log
I20260806 15:54:35.196367 26813 ts_tablet_manager.cc:1428] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:35.196396 26812 ts_tablet_manager.cc:1397] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260806 15:54:35.196841 26812 raft_consensus.cc:348] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.196964 26812 raft_consensus.cc:374] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.197006 26812 raft_consensus.cc:729] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a816b1c765548bf9594026a42f90530, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.197153 26812 consensus_queue.cc:260] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.197468 26812 ts_tablet_manager.cc:1428] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:54:35.197433 26834 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.197563 26834 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.336192 26817 raft_consensus.cc:482] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:35.336347 26817 raft_consensus.cc:504] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.336881 26817 leader_election.cc:288] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:35.337419 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39a7cb45f594a5785a7b69367a11367" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0" is_pre_election: true
I20260806 15:54:35.337430 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39a7cb45f594a5785a7b69367a11367" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530" is_pre_election: true
I20260806 15:54:35.337637 26491 raft_consensus.cc:2454] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 0.
I20260806 15:54:35.337637 26277 raft_consensus.cc:2454] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 0.
I20260806 15:54:35.337958 26594 leader_election.cc:302] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a816b1c765548bf9594026a42f90530, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.338155 26817 raft_consensus.cc:2781] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:35.338217 26817 raft_consensus.cc:482] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:35.338304 26817 raft_consensus.cc:3037] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.339285 26817 raft_consensus.cc:504] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.339666 26817 leader_election.cc:288] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 election: Requested vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:35.339999 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39a7cb45f594a5785a7b69367a11367" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530"
I20260806 15:54:35.340059 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39a7cb45f594a5785a7b69367a11367" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0"
I20260806 15:54:35.340127 26277 raft_consensus.cc:3037] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.340195 26491 raft_consensus.cc:3037] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.341164 26491 raft_consensus.cc:2454] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 1.
I20260806 15:54:35.341199 26277 raft_consensus.cc:2454] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 1.
I20260806 15:54:35.341465 26595 leader_election.cc:302] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.341681 26817 raft_consensus.cc:2781] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:35.341792 26817 raft_consensus.cc:686] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [term 1 LEADER]: Becoming Leader. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Running, Role: LEADER
I20260806 15:54:35.341929 26817 consensus_queue.cc:237] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } }
I20260806 15:54:35.342607 26852 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: TxnStatusTablet state changed. Reason: New leader b071c370dbf141be97fe0948ffeca35d. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.342720 26852 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:35.342949 26870 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 15:54:35.343156 26870 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 15:54:35.343891 26026 catalog_manager.cc:5495] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d reported cstate change: term changed from 0 to 1, leader changed from <none> to b071c370dbf141be97fe0948ffeca35d (127.13.223.3). New cstate: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:35.391522 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.386908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40751 (local address 127.0.0.1:42430)
0806 15:54:35.387153 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.387171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.387289 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.387688 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.387691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.387700 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:35.387814 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.387819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.388782 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.388788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.389800 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.389811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.390289 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.390293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.390389 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.390437 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.391188 (+ 751us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.391206 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.391273 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.391405 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 15:54:35.391537 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.387023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.13.223.3:40751)
0806 15:54:35.387304 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.387308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.387342 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:54:35.387485 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.387488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.387546 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.387675 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:35.387930 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.388625 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.389942 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.390164 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.390276 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.390644 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.391074 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.391203 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.391284 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.391363 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
I20260806 15:54:35.394923 26491 raft_consensus.cc:1261] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Refusing update from remote peer b071c370dbf141be97fe0948ffeca35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.395251 26277 raft_consensus.cc:1261] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Refusing update from remote peer b071c370dbf141be97fe0948ffeca35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.395601 26852 consensus_queue.cc:1035] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.395789 26817 consensus_queue.cc:1035] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.397451 26819 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: TxnStatusTablet state changed. Reason: New leader b071c370dbf141be97fe0948ffeca35d. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.397593 26819 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.397598 26834 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: TxnStatusTablet state changed. Reason: New leader b071c370dbf141be97fe0948ffeca35d. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.397715 26834 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.400539 26817 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.400548 26852 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.400705 26817 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:35.400710 26852 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P b071c370dbf141be97fe0948ffeca35d: This TxnStatusTablet replica's current role is: LEADER
I20260806 15:54:35.401010 26834 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.401111 26834 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P 9a816b1c765548bf9594026a42f90530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.401845 26819 tablet_replica.cc:439] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } }
I20260806 15:54:35.401952 26819 tablet_replica.cc:442] T f39a7cb45f594a5785a7b69367a11367 P 43eb9cc094014655b16fe4d6df8083a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 15:54:35.405287 26880 mvcc.cc:205] Tried to move back new op lower bound from 7315585742412738560 to 7315585742201946112. Current Snapshot: MvccSnapshot[applied={T|T < 7315585742412738560}]
I20260806 15:54:35.410393 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.406745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.13.223.3:40751)
0806 15:54:35.406865 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.406869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.407015 (+ 146us) server_negotiation.cc:407] Connection header received
0806 15:54:35.407129 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.407132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.407212 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.407292 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:35.407594 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.408364 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.409282 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.409640 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.409750 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.409978 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.410058 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.410129 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.410208 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.410261 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:54:35.424109 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.418264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:56853 (local address 127.13.223.3:40751)
0806 15:54:35.418381 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.418384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.420917 (+ 2533us) server_negotiation.cc:407] Connection header received
0806 15:54:35.420967 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.420969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.421009 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.421085 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:35.421384 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.421863 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.423087 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.423287 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.423364 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.423687 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.423747 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.423800 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.423940 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.423981 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 15:54:35.424168 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.418177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40751 (local address 127.13.223.3:56853)
0806 15:54:35.420736 (+ 2559us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.420751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.420877 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.421119 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.421122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.421141 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 15:54:35.421271 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.421277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.421971 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.421975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.422961 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.422972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.423398 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.423402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.423498 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.423550 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.423833 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.423839 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.423917 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.423990 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2443,"thread_start_us":101,"threads_started":1}
I20260806 15:54:35.431061 26491 raft_consensus.cc:1261] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Refusing update from remote peer b071c370dbf141be97fe0948ffeca35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.431066 26277 raft_consensus.cc:1261] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Refusing update from remote peer b071c370dbf141be97fe0948ffeca35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.431492 26817 consensus_queue.cc:1035] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.431643 26852 consensus_queue.cc:1035] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.480584 26026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:35.480777 26026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:35.483148 26026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Sending DeleteTablet for 3 replicas of tablet 24eb99d9ad004d4cb06cfa9cfed42264
I20260806 15:54:35.483805 26244 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eb99d9ad004d4cb06cfa9cfed42264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33354
I20260806 15:54:35.483840 26461 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eb99d9ad004d4cb06cfa9cfed42264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40316
I20260806 15:54:35.484246 26901 tablet_replica.cc:330] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: stopping tablet replica
I20260806 15:54:35.484413 26901 raft_consensus.cc:2229] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:35.484597 26902 tablet_replica.cc:330] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: stopping tablet replica
I20260806 15:54:35.484622 26901 raft_consensus.cc:2258] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.484841 26902 raft_consensus.cc:2229] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:35.484855 26670 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eb99d9ad004d4cb06cfa9cfed42264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:42428
I20260806 15:54:35.485095 26902 raft_consensus.cc:2258] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.485612 26903 tablet_replica.cc:330] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: stopping tablet replica
I20260806 15:54:35.485798 26903 raft_consensus.cc:2229] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:35.486088 26903 raft_consensus.cc:2258] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.486088 26902 ts_tablet_manager.cc:1905] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.486922 26903 ts_tablet_manager.cc:1905] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.489825 26901 ts_tablet_manager.cc:1905] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.489989 26902 ts_tablet_manager.cc:1918] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:35.490070 26903 ts_tablet_manager.cc:1918] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:35.490165 26903 log.cc:1196] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/24eb99d9ad004d4cb06cfa9cfed42264
I20260806 15:54:35.490185 26902 log.cc:1196] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/24eb99d9ad004d4cb06cfa9cfed42264
I20260806 15:54:35.490476 26903 ts_tablet_manager.cc:1939] T 24eb99d9ad004d4cb06cfa9cfed42264 P b071c370dbf141be97fe0948ffeca35d: Deleting consensus metadata
I20260806 15:54:35.490710 26902 ts_tablet_manager.cc:1939] T 24eb99d9ad004d4cb06cfa9cfed42264 P 43eb9cc094014655b16fe4d6df8083a0: Deleting consensus metadata
I20260806 15:54:35.491262 26007 catalog_manager.cc:4841] TS b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751): tablet 24eb99d9ad004d4cb06cfa9cfed42264 (table test-table [id=0b77470eef3347a89dcaa1314103aab0]) successfully deleted
I20260806 15:54:35.491504 26010 catalog_manager.cc:4841] TS 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201): tablet 24eb99d9ad004d4cb06cfa9cfed42264 (table test-table [id=0b77470eef3347a89dcaa1314103aab0]) successfully deleted
I20260806 15:54:35.492905 26901 ts_tablet_manager.cc:1918] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:35.492985 26901 log.cc:1196] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/24eb99d9ad004d4cb06cfa9cfed42264
I20260806 15:54:35.493273 26901 ts_tablet_manager.cc:1939] T 24eb99d9ad004d4cb06cfa9cfed42264 P 9a816b1c765548bf9594026a42f90530: Deleting consensus metadata
I20260806 15:54:35.494050 26009 catalog_manager.cc:4841] TS 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947): tablet 24eb99d9ad004d4cb06cfa9cfed42264 (table test-table [id=0b77470eef3347a89dcaa1314103aab0]) successfully deleted
Aug 06 15:54:35 dist-test-slave-rwrg krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 15:54:35 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:35 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 15:54:35.520959 26863 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:35.514339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.13.223.62:34829)
0806 15:54:35.514502 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.514507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.514702 (+ 195us) server_negotiation.cc:407] Connection header received
0806 15:54:35.514896 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.514899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.514949 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.515047 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:35.515871 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.516431 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.517355 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.517597 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.517706 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.520817 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44712: BlockingRecv error: recv got EOF from 127.0.0.1:44712 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 15:54:35 dist-test-slave-rwrg krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM
I20260806 15:54:35.531785 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.522614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.13.223.62:34829)
0806 15:54:35.522734 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.522738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.522832 (+ 94us) server_negotiation.cc:407] Connection header received
0806 15:54:35.522929 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.522932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.522981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.523072 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:35.523626 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.524376 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.525269 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.525621 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.525734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.528297 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:35.528302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:35.528305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:35.528334 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:35.530050 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:35.530485 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:35.530489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:35.530491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:35.530595 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:35.530888 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:35.530892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:35.530894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:35.531119 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:35.531253 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.531473 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.531613 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:54:35.534970 26026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:54:35.535435 26026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:35.540464 26244 tablet_service.cc:1467] Processing CreateTablet for tablet 0ccaac78421d48638253b4aee5054085 (DEFAULT_TABLE table=test-table [id=8b31ac6078f24f0bba0249b45c4fbc5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:35.540768 26461 tablet_service.cc:1467] Processing CreateTablet for tablet 0ccaac78421d48638253b4aee5054085 (DEFAULT_TABLE table=test-table [id=8b31ac6078f24f0bba0249b45c4fbc5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:35.540768 26670 tablet_service.cc:1467] Processing CreateTablet for tablet 0ccaac78421d48638253b4aee5054085 (DEFAULT_TABLE table=test-table [id=8b31ac6078f24f0bba0249b45c4fbc5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:35.540849 26244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ccaac78421d48638253b4aee5054085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.541049 26670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ccaac78421d48638253b4aee5054085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.541055 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ccaac78421d48638253b4aee5054085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.543574 26812 tablet_bootstrap.cc:492] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Bootstrap starting.
I20260806 15:54:35.543609 26810 tablet_bootstrap.cc:492] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Bootstrap starting.
I20260806 15:54:35.543934 26813 tablet_bootstrap.cc:492] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Bootstrap starting.
I20260806 15:54:35.544436 26812 tablet_bootstrap.cc:654] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.544461 26810 tablet_bootstrap.cc:654] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.544891 26813 tablet_bootstrap.cc:654] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.545912 26812 tablet_bootstrap.cc:492] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: No bootstrap required, opened a new log
I20260806 15:54:35.546008 26812 ts_tablet_manager.cc:1397] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:35.546407 26812 raft_consensus.cc:348] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.546535 26812 raft_consensus.cc:374] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.546577 26812 raft_consensus.cc:729] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a816b1c765548bf9594026a42f90530, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.546703 26810 tablet_bootstrap.cc:492] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: No bootstrap required, opened a new log
I20260806 15:54:35.546700 26812 consensus_queue.cc:260] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.546800 26810 ts_tablet_manager.cc:1397] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:35.546957 26812 ts_tablet_manager.cc:1428] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:35.547211 26810 raft_consensus.cc:348] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.547329 26810 raft_consensus.cc:374] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.547369 26810 raft_consensus.cc:729] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.547520 26810 consensus_queue.cc:260] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.547832 26810 ts_tablet_manager.cc:1428] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:35.548326 26813 tablet_bootstrap.cc:492] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: No bootstrap required, opened a new log
I20260806 15:54:35.548419 26813 ts_tablet_manager.cc:1397] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260806 15:54:35.549259 26813 raft_consensus.cc:348] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.549415 26813 raft_consensus.cc:374] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.549463 26813 raft_consensus.cc:729] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eb9cc094014655b16fe4d6df8083a0, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.549592 26813 consensus_queue.cc:260] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.549881 26813 ts_tablet_manager.cc:1428] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 15:54:35.630303 26819 raft_consensus.cc:482] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:35.630491 26819 raft_consensus.cc:504] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.631825 26819 leader_election.cc:288] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:35.637244 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.632276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:51551 (local address 127.13.223.3:40751)
0806 15:54:35.632425 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.632432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.632937 (+ 505us) server_negotiation.cc:407] Connection header received
0806 15:54:35.633067 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.633071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.633130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.633206 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:35.633633 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.634403 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.635969 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.636427 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.636624 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.636692 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.637017 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.637114 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 15:54:35.637275 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.632024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45947 (local address 127.13.223.2:51661)
0806 15:54:35.632395 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.632413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.632554 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.633208 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.633211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.633233 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:35.633343 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.633351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.634254 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.634260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.635665 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.635676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.636695 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.636700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.636892 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.636926 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.636999 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.637054 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":58,"thread_start_us":113,"threads_started":1}
I20260806 15:54:35.637275 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.632129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40751 (local address 127.13.223.2:51551)
0806 15:54:35.632807 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:35.632822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:35.632942 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:35.633327 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:35.633330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:35.633348 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:54:35.633489 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.633496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.634563 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.634567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.635831 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:35.635839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.636608 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.636613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:35.636807 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.636835 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 15:54:35.636965 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 15:54:35.637053 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":27,"thread_start_us":122,"threads_started":1}
I20260806 15:54:35.637655 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.632628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:51661 (local address 127.13.223.1:45947)
0806 15:54:35.632868 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.632871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.632913 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:54:35.632985 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.632988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.633053 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.633176 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:54:35.633496 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.634124 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.635817 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.636481 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.636800 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.636854 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.637344 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.637448 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260806 15:54:35.638264 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ccaac78421d48638253b4aee5054085" candidate_uuid: "43eb9cc094014655b16fe4d6df8083a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b071c370dbf141be97fe0948ffeca35d" is_pre_election: true
I20260806 15:54:35.638273 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ccaac78421d48638253b4aee5054085" candidate_uuid: "43eb9cc094014655b16fe4d6df8083a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530" is_pre_election: true
I20260806 15:54:35.638445 26277 raft_consensus.cc:2454] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43eb9cc094014655b16fe4d6df8083a0 in term 0.
I20260806 15:54:35.638447 26701 raft_consensus.cc:2454] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43eb9cc094014655b16fe4d6df8083a0 in term 0.
I20260806 15:54:35.638860 26383 leader_election.cc:302] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, 9a816b1c765548bf9594026a42f90530; no voters:
I20260806 15:54:35.639194 26819 raft_consensus.cc:2781] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:35.639269 26819 raft_consensus.cc:482] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:35.639312 26819 raft_consensus.cc:3037] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.640728 26819 raft_consensus.cc:504] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.641160 26819 leader_election.cc:288] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [CANDIDATE]: Term 1 election: Requested vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751)
I20260806 15:54:35.641569 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ccaac78421d48638253b4aee5054085" candidate_uuid: "43eb9cc094014655b16fe4d6df8083a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530"
I20260806 15:54:35.641569 26701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ccaac78421d48638253b4aee5054085" candidate_uuid: "43eb9cc094014655b16fe4d6df8083a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b071c370dbf141be97fe0948ffeca35d"
I20260806 15:54:35.641709 26277 raft_consensus.cc:3037] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.641732 26701 raft_consensus.cc:3037] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.642781 26701 raft_consensus.cc:2454] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43eb9cc094014655b16fe4d6df8083a0 in term 1.
I20260806 15:54:35.642951 26277 raft_consensus.cc:2454] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43eb9cc094014655b16fe4d6df8083a0 in term 1.
I20260806 15:54:35.643266 26381 leader_election.cc:302] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.643476 26819 raft_consensus.cc:2781] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:35.643736 26819 raft_consensus.cc:686] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 LEADER]: Becoming Leader. State: Replica: 43eb9cc094014655b16fe4d6df8083a0, State: Running, Role: LEADER
I20260806 15:54:35.644016 26819 consensus_queue.cc:237] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.646834 26029 catalog_manager.cc:5495] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "43eb9cc094014655b16fe4d6df8083a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
I20260806 15:54:35.678613 26936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.674150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.13.223.2:39201)
0806 15:54:35.674444 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.674451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.674466 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:35.674561 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.674563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.674625 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.674743 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:35.675021 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.675614 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.676864 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.677114 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.677215 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.677558 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.678074 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.678171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.678278 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.678332 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
I20260806 15:54:35.681595 26701 raft_consensus.cc:1261] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Refusing update from remote peer 43eb9cc094014655b16fe4d6df8083a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.681695 26277 raft_consensus.cc:1261] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Refusing update from remote peer 43eb9cc094014655b16fe4d6df8083a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.682189 26932 consensus_queue.cc:1035] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.682401 26819 consensus_queue.cc:1035] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.695971 26028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:35.696143 26028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:35.698434 26028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Sending DeleteTablet for 3 replicas of tablet 0ccaac78421d48638253b4aee5054085
I20260806 15:54:35.699092 26670 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ccaac78421d48638253b4aee5054085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:42428
I20260806 15:54:35.699115 26244 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ccaac78421d48638253b4aee5054085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33354
I20260806 15:54:35.699311 26903 tablet_replica.cc:330] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: stopping tablet replica
I20260806 15:54:35.699445 26903 raft_consensus.cc:2229] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:35.699594 26903 raft_consensus.cc:2258] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.699790 26901 tablet_replica.cc:330] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: stopping tablet replica
I20260806 15:54:35.699924 26901 raft_consensus.cc:2229] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:35.700067 26901 raft_consensus.cc:2258] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.700771 26901 ts_tablet_manager.cc:1905] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.700789 26461 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ccaac78421d48638253b4aee5054085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40316
I20260806 15:54:35.700970 26902 tablet_replica.cc:330] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: stopping tablet replica
I20260806 15:54:35.701098 26902 raft_consensus.cc:2229] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:35.701373 26902 raft_consensus.cc:2258] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.701946 26029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 15:54:35.702042 26902 ts_tablet_manager.cc:1905] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 15:54:35.702349 26029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:54:35.703451 26901 ts_tablet_manager.cc:1918] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:35.703526 26901 log.cc:1196] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/0ccaac78421d48638253b4aee5054085
I20260806 15:54:35.704272 26901 ts_tablet_manager.cc:1939] T 0ccaac78421d48638253b4aee5054085 P 9a816b1c765548bf9594026a42f90530: Deleting consensus metadata
I20260806 15:54:35.704638 26902 ts_tablet_manager.cc:1918] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:35.704718 26902 log.cc:1196] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/0ccaac78421d48638253b4aee5054085
I20260806 15:54:35.705206 26902 ts_tablet_manager.cc:1939] T 0ccaac78421d48638253b4aee5054085 P 43eb9cc094014655b16fe4d6df8083a0: Deleting consensus metadata
I20260806 15:54:35.705366 26009 catalog_manager.cc:4841] TS 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947): tablet 0ccaac78421d48638253b4aee5054085 (table test-table [id=8b31ac6078f24f0bba0249b45c4fbc5c]) successfully deleted
I20260806 15:54:35.705904 26010 catalog_manager.cc:4841] TS 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201): tablet 0ccaac78421d48638253b4aee5054085 (table test-table [id=8b31ac6078f24f0bba0249b45c4fbc5c]) successfully deleted
I20260806 15:54:35.710209 26903 ts_tablet_manager.cc:1905] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.714087 26903 ts_tablet_manager.cc:1918] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:54:35.714193 26903 log.cc:1196] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/0ccaac78421d48638253b4aee5054085
I20260806 15:54:35.714458 26903 ts_tablet_manager.cc:1939] T 0ccaac78421d48638253b4aee5054085 P b071c370dbf141be97fe0948ffeca35d: Deleting consensus metadata
I20260806 15:54:35.714982 26461 tablet_service.cc:1467] Processing CreateTablet for tablet b350251682ec40dab3d043ad32d2904c (DEFAULT_TABLE table=test-table [id=caa39cf8cd394d33be8edc8dff3b879a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:35.715087 26244 tablet_service.cc:1467] Processing CreateTablet for tablet b350251682ec40dab3d043ad32d2904c (DEFAULT_TABLE table=test-table [id=caa39cf8cd394d33be8edc8dff3b879a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:35.715268 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b350251682ec40dab3d043ad32d2904c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.715379 26244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b350251682ec40dab3d043ad32d2904c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.715320 26670 tablet_service.cc:1467] Processing CreateTablet for tablet b350251682ec40dab3d043ad32d2904c (DEFAULT_TABLE table=test-table [id=caa39cf8cd394d33be8edc8dff3b879a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:54:35.715551 26670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b350251682ec40dab3d043ad32d2904c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:35.717563 26813 tablet_bootstrap.cc:492] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Bootstrap starting.
I20260806 15:54:35.718479 26813 tablet_bootstrap.cc:654] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.719123 26812 tablet_bootstrap.cc:492] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Bootstrap starting.
I20260806 15:54:35.719717 26813 tablet_bootstrap.cc:492] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: No bootstrap required, opened a new log
I20260806 15:54:35.719830 26813 ts_tablet_manager.cc:1397] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:35.719907 26812 tablet_bootstrap.cc:654] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.720284 26813 raft_consensus.cc:348] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.720431 26813 raft_consensus.cc:374] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.720432 26810 tablet_bootstrap.cc:492] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Bootstrap starting.
I20260806 15:54:35.720474 26813 raft_consensus.cc:729] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eb9cc094014655b16fe4d6df8083a0, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.720645 26813 consensus_queue.cc:260] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.720963 26813 ts_tablet_manager.cc:1428] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 15:54:35.721248 26810 tablet_bootstrap.cc:654] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:35.722595 26810 tablet_bootstrap.cc:492] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: No bootstrap required, opened a new log
I20260806 15:54:35.722685 26810 ts_tablet_manager.cc:1397] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 15:54:35.723084 26810 raft_consensus.cc:348] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.723219 26810 raft_consensus.cc:374] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.723263 26810 raft_consensus.cc:729] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.723456 26812 tablet_bootstrap.cc:492] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: No bootstrap required, opened a new log
I20260806 15:54:35.723425 26810 consensus_queue.cc:260] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.723549 26812 ts_tablet_manager.cc:1397] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 15:54:35.723704 26810 ts_tablet_manager.cc:1428] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:54:35.723927 26812 raft_consensus.cc:348] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.724043 26812 raft_consensus.cc:374] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:35.724081 26812 raft_consensus.cc:729] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a816b1c765548bf9594026a42f90530, State: Initialized, Role: FOLLOWER
I20260806 15:54:35.724218 26812 consensus_queue.cc:260] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.724524 26812 ts_tablet_manager.cc:1428] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:54:35.726295 26007 catalog_manager.cc:4841] TS b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751): tablet 0ccaac78421d48638253b4aee5054085 (table test-table [id=8b31ac6078f24f0bba0249b45c4fbc5c]) successfully deleted
I20260806 15:54:35.902279 26852 raft_consensus.cc:482] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:54:35.902462 26852 raft_consensus.cc:504] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.903026 26852 leader_election.cc:288] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:35.903522 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350251682ec40dab3d043ad32d2904c" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530" is_pre_election: true
I20260806 15:54:35.903549 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350251682ec40dab3d043ad32d2904c" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0" is_pre_election: true
I20260806 15:54:35.903697 26277 raft_consensus.cc:2454] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 0.
I20260806 15:54:35.903786 26491 raft_consensus.cc:2454] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 0.
I20260806 15:54:35.904094 26594 leader_election.cc:302] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a816b1c765548bf9594026a42f90530, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.904279 26852 raft_consensus.cc:2781] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:54:35.904354 26852 raft_consensus.cc:482] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:54:35.904409 26852 raft_consensus.cc:3037] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.905481 26852 raft_consensus.cc:504] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.905825 26852 leader_election.cc:288] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 election: Requested vote from peers 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947), 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201)
I20260806 15:54:35.906265 26491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350251682ec40dab3d043ad32d2904c" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eb9cc094014655b16fe4d6df8083a0"
I20260806 15:54:35.906260 26277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b350251682ec40dab3d043ad32d2904c" candidate_uuid: "b071c370dbf141be97fe0948ffeca35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a816b1c765548bf9594026a42f90530"
I20260806 15:54:35.906405 26491 raft_consensus.cc:3037] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.906424 26277 raft_consensus.cc:3037] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:35.907433 26491 raft_consensus.cc:2454] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 1.
I20260806 15:54:35.907514 26277 raft_consensus.cc:2454] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b071c370dbf141be97fe0948ffeca35d in term 1.
I20260806 15:54:35.907728 26595 leader_election.cc:302] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eb9cc094014655b16fe4d6df8083a0, b071c370dbf141be97fe0948ffeca35d; no voters:
I20260806 15:54:35.907920 26852 raft_consensus.cc:2781] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:35.908015 26852 raft_consensus.cc:686] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 1 LEADER]: Becoming Leader. State: Replica: b071c370dbf141be97fe0948ffeca35d, State: Running, Role: LEADER
I20260806 15:54:35.908157 26852 consensus_queue.cc:237] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } }
I20260806 15:54:35.909993 26026 catalog_manager.cc:5495] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d reported cstate change: term changed from 0 to 1, leader changed from <none> to b071c370dbf141be97fe0948ffeca35d (127.13.223.3). New cstate: current_term: 1 leader_uuid: "b071c370dbf141be97fe0948ffeca35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b071c370dbf141be97fe0948ffeca35d" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40751 } health_report { overall_health: HEALTHY } } }
I20260806 15:54:35.927373 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:35.924625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.13.223.3:40751)
0806 15:54:35.924765 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:35.924768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:35.924801 (+ 33us) server_negotiation.cc:407] Connection header received
0806 15:54:35.924967 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:35.924969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:35.925009 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:35.925069 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:54:35.925274 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.925784 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.926541 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:35.926758 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:35.926832 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:35.927038 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:54:35.927101 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:54:35.927153 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:35.927231 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 15:54:35.927273 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 15:54:35.933952 26277 raft_consensus.cc:1261] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Refusing update from remote peer b071c370dbf141be97fe0948ffeca35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.933951 26491 raft_consensus.cc:1261] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Refusing update from remote peer b071c370dbf141be97fe0948ffeca35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:54:35.934348 26852 consensus_queue.cc:1035] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eb9cc094014655b16fe4d6df8083a0" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.934556 26817 consensus_queue.cc:1035] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a816b1c765548bf9594026a42f90530" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:54:35.963466 26028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:35.963617 26028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:54:35.965541 26028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32a946d5a3a4a74a93d2c47501415b0: Sending DeleteTablet for 3 replicas of tablet b350251682ec40dab3d043ad32d2904c
I20260806 15:54:35.966065 26461 tablet_service.cc:1514] Processing DeleteTablet for tablet b350251682ec40dab3d043ad32d2904c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40316
I20260806 15:54:35.966076 26244 tablet_service.cc:1514] Processing DeleteTablet for tablet b350251682ec40dab3d043ad32d2904c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33354
I20260806 15:54:35.966141 26670 tablet_service.cc:1514] Processing DeleteTablet for tablet b350251682ec40dab3d043ad32d2904c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:54:35 UTC) from {username='oryx', principal='oryx/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:42428
I20260806 15:54:35.966260 26902 tablet_replica.cc:330] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: stopping tablet replica
I20260806 15:54:35.966274 26901 tablet_replica.cc:330] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: stopping tablet replica
I20260806 15:54:35.966401 26901 raft_consensus.cc:2229] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:35.966401 26902 raft_consensus.cc:2229] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:54:35.966531 26901 raft_consensus.cc:2258] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.966544 26902 raft_consensus.cc:2258] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.966738 26903 tablet_replica.cc:330] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: stopping tablet replica
I20260806 15:54:35.966873 26903 raft_consensus.cc:2229] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:54:35.967090 26903 raft_consensus.cc:2258] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:54:35.967338 26901 ts_tablet_manager.cc:1905] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.967782 26902 ts_tablet_manager.cc:1905] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.967787 26903 ts_tablet_manager.cc:1905] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:54:35.970137 26901 ts_tablet_manager.cc:1918] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:35.970228 26901 log.cc:1196] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/b350251682ec40dab3d043ad32d2904c
I20260806 15:54:35.970518 26901 ts_tablet_manager.cc:1939] T b350251682ec40dab3d043ad32d2904c P 9a816b1c765548bf9594026a42f90530: Deleting consensus metadata
I20260806 15:54:35.970620 26903 ts_tablet_manager.cc:1918] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:35.970700 26903 log.cc:1196] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/b350251682ec40dab3d043ad32d2904c
I20260806 15:54:35.970964 26903 ts_tablet_manager.cc:1939] T b350251682ec40dab3d043ad32d2904c P b071c370dbf141be97fe0948ffeca35d: Deleting consensus metadata
I20260806 15:54:35.971298 26902 ts_tablet_manager.cc:1918] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 15:54:35.971376 26902 log.cc:1196] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/b350251682ec40dab3d043ad32d2904c
I20260806 15:54:35.971360 26009 catalog_manager.cc:4841] TS 9a816b1c765548bf9594026a42f90530 (127.13.223.1:45947): tablet b350251682ec40dab3d043ad32d2904c (table test-table [id=caa39cf8cd394d33be8edc8dff3b879a]) successfully deleted
I20260806 15:54:35.971645 26902 ts_tablet_manager.cc:1939] T b350251682ec40dab3d043ad32d2904c P 43eb9cc094014655b16fe4d6df8083a0: Deleting consensus metadata
I20260806 15:54:35.971644 26007 catalog_manager.cc:4841] TS b071c370dbf141be97fe0948ffeca35d (127.13.223.3:40751): tablet b350251682ec40dab3d043ad32d2904c (table test-table [id=caa39cf8cd394d33be8edc8dff3b879a]) successfully deleted
I20260806 15:54:35.972122 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 26149
I20260806 15:54:35.981221 26010 catalog_manager.cc:4841] TS 43eb9cc094014655b16fe4d6df8083a0 (127.13.223.2:39201): tablet b350251682ec40dab3d043ad32d2904c (table test-table [id=caa39cf8cd394d33be8edc8dff3b879a]) successfully deleted
I20260806 15:54:35.987380 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 26360
I20260806 15:54:36.004850 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 26569
I20260806 15:54:36.020401 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 25989
2026-08-06T15:54:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3925 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:36 dist-test-slave-rwrg krb5kdc[26960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:36 dist-test-slave-rwrg krb5kdc[26960](info): set up 2 sockets
Aug 06 15:54:36 dist-test-slave-rwrg krb5kdc[26960](info): commencing operation
krb5kdc: starting...
W20260806 15:54:37.339165 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:37 dist-test-slave-rwrg krb5kdc[26960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:37.472259 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40985
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:40985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:37.589401 26975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:37.589690 26975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:37.589731 26975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:37.593835 26975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:37.593922 26975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:37.593942 26975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:37.593958 26975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:37.593973 26975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:37.599617 26975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:40985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:37.600937 26975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:37.602008 26975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:37.610833 26982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:37.610795 26983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:37.610841 26985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:37.610807 26975 server_base.cc:1034] running on GCE node
I20260806 15:54:37.611701 26975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:37.612968 26975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:37.614174 26975 hybrid_clock.cc:648] HybridClock initialized: now 1786031677614158 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T15:54:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1600 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:37 dist-test-slave-rwrg krb5kdc[26994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:37 dist-test-slave-rwrg krb5kdc[26994](info): set up 2 sockets
Aug 06 15:54:37 dist-test-slave-rwrg krb5kdc[26994](info): commencing operation
krb5kdc: starting...
W20260806 15:54:38.971678 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:39.101627 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:45197
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:39.217768 27009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:39.218035 27009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:39.218077 27009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:39.222019 27009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:39.222082 27009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:39.222102 27009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:39.222118 27009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:39.222133 27009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:39.227454 27009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:39.228578 27009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:39.229592 27009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:39.238528 27022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:39.238544 27019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:39.238583 27009 server_base.cc:1034] running on GCE node
W20260806 15:54:39.238534 27024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:39.239339 27009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:39.240410 27009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:39.241611 27009 hybrid_clock.cc:648] HybridClock initialized: now 1786031679241580 us; error 44 us; skew 500 ppm
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:39.244426 27009 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:39.245428 27009 webserver.cc:460] Webserver started at http://127.13.223.62:39457/ using document root <none> and password file <none>
I20260806 15:54:39.246006 27009 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:39.246105 27009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:39.246354 27009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:39.248448 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "9964a6affa69490f9433fb8511d79686"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "dd696f1b63a62519d5868268bf2b9676"
server_key_iv: "4ce6b6c259603c852eb55d8be6ec4b5e"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.249095 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "9964a6affa69490f9433fb8511d79686"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "dd696f1b63a62519d5868268bf2b9676"
server_key_iv: "4ce6b6c259603c852eb55d8be6ec4b5e"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.252972 27009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 15:54:39.255378 27037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.256747 27009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:39.256894 27009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "9964a6affa69490f9433fb8511d79686"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "dd696f1b63a62519d5868268bf2b9676"
server_key_iv: "4ce6b6c259603c852eb55d8be6ec4b5e"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.257027 27009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:39.281771 27009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:39.284984 27009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:39.285218 27009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:39.292778 27009 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:45197
I20260806 15:54:39.292809 27113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:45197 every 8 connection(s)
I20260806 15:54:39.293777 27009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:39.296589 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:39.298977 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 27009
I20260806 15:54:39.299081 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:39.299384 14204 external_mini_cluster.cc:1428] Setting key f7434531498c0f33ffaca8429501bc5c
I20260806 15:54:39.303076 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Bootstrap starting.
I20260806 15:54:39.305904 27114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:39.306754 27114 log.cc:824] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:39.309020 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: No bootstrap required, opened a new log
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:39.312850 27114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:39.313165 27114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:39.313237 27114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964a6affa69490f9433fb8511d79686, State: Initialized, Role: FOLLOWER
I20260806 15:54:39.313750 27114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:39.313951 27114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:39.314042 27114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:39.314191 27114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:39.315412 27114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:39.315761 27114 leader_election.cc:302] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9964a6affa69490f9433fb8511d79686; no voters:
I20260806 15:54:39.316121 27114 leader_election.cc:288] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:39.316210 27121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:39.316404 27121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 LEADER]: Becoming Leader. State: Replica: 9964a6affa69490f9433fb8511d79686, State: Running, Role: LEADER
I20260806 15:54:39.316476 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:39.300981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.13.223.62:45197)
0806 15:54:39.301452 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:39.301472 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:39.301519 (+ 47us) server_negotiation.cc:407] Connection header received
0806 15:54:39.302347 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:39.302372 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:39.302799 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:39.303189 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:39.304075 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.305672 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.306627 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.307308 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.307428 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:39.310455 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:39.310461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:39.310476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:39.310525 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:39.312979 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:39.313578 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:39.313586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:39.313593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:39.313726 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:39.314049 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:39.314054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:39.314059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:39.314603 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:39.314780 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:39.315071 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 15:54:39.315398 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":126,"threads_started":1}
I20260806 15:54:39.316731 27121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:39.317507 27114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:39.319339 27122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9964a6affa69490f9433fb8511d79686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } } }
I20260806 15:54:39.319483 27122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:39.319850 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9964a6affa69490f9433fb8511d79686. Latest consensus state: current_term: 1 leader_uuid: "9964a6affa69490f9433fb8511d79686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } } }
I20260806 15:54:39.320165 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: This master's current role is: LEADER
W20260806 15:54:39.322686 27147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:54:39.322777 27147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:54:39.322856 27146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:39.323565 27146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:39.329051 27146 catalog_manager.cc:1347] Generated new cluster ID: ad6c57f2053f4428997ed108515b7b58
I20260806 15:54:39.329128 27146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:39.336618 27146 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:39.337916 27146 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:39.344136 27146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Generated new TSK 0
I20260806 15:54:39.344918 27146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:39.379142 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:45197
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:39.523500 27154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:39.523761 27154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:39.523804 27154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:39.527841 27154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:39.527901 27154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:39.527957 27154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:39.533125 27154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:39.534154 27154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:39.535144 27154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:39.544994 27166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:39.545184 27154 server_base.cc:1034] running on GCE node
W20260806 15:54:39.545018 27167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:39.544983 27169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:39.545719 27154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:39.546432 27154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:39.547660 27154 hybrid_clock.cc:648] HybridClock initialized: now 1786031679547644 us; error 28 us; skew 500 ppm
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:39.550647 27154 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:39.551574 27154 webserver.cc:460] Webserver started at http://127.13.223.1:36555/ using document root <none> and password file <none>
I20260806 15:54:39.552158 27154 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:39.552253 27154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:39.552529 27154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:39.554697 27154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "73824ea48f6346e682d73d5451b817b8"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "f996e391c54aff822b3a23db9e146a76"
server_key_iv: "127829d4e867236f36f361c453926138"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.555315 27154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "73824ea48f6346e682d73d5451b817b8"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "f996e391c54aff822b3a23db9e146a76"
server_key_iv: "127829d4e867236f36f361c453926138"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.559386 27154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:54:39.562297 27180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.563652 27154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 15:54:39.563802 27154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "73824ea48f6346e682d73d5451b817b8"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "f996e391c54aff822b3a23db9e146a76"
server_key_iv: "127829d4e867236f36f361c453926138"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.563922 27154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:39.593921 27154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:39.596963 27154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:39.597208 27154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:39.597851 27154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:39.598738 27154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:39.598814 27154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.598888 27154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:39.598932 27154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.608860 27154 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:43843
I20260806 15:54:39.608887 27358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:43843 every 8 connection(s)
I20260806 15:54:39.609848 27154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:39.617743 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 27154
I20260806 15:54:39.617935 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:39.618299 14204 external_mini_cluster.cc:1428] Setting key d3bcc9bbef60d5a8011009f1b43e405c
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:39.626405 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:39.611715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:60425 (local address 127.13.223.62:45197)
0806 15:54:39.611910 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:39.611914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:39.613127 (+ 1213us) server_negotiation.cc:407] Connection header received
0806 15:54:39.614228 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:39.614232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:39.614287 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:39.614379 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:39.615904 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.616483 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.618157 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.618498 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.618631 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:39.622074 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:39.622081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:39.622087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:39.622124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:39.623777 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:39.624576 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:39.624580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:39.624581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:39.624663 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:39.625105 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:39.625107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:39.625108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:39.625287 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:39.625360 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:39.626160 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 15:54:39.626280 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 15:54:39.627873 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:39.612104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:60425)
0806 15:54:39.612940 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:39.612989 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:39.613965 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:39.614627 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:39.614635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:39.615053 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:39.615674 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:39.615692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.616709 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.616713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:39.617858 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:39.617897 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.618741 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.618747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:39.618989 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:39.619738 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:39.619763 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:39.621857 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:39.623911 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:39.623921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:39.623943 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:39.624374 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:39.624764 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:39.624770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:39.624776 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:39.624997 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:39.625412 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:39.625424 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:39.625841 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 15:54:39.626103 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 15:54:39.626537 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":115,"threads_started":1}
I20260806 15:54:39.629352 27363 heartbeater.cc:347] Connected to a master server at 127.13.223.62:45197
I20260806 15:54:39.629662 27363 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:39.630345 27363 heartbeater.cc:511] Master 127.13.223.62:45197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:54:39.632263 27058 ts_manager.cc:194] Registered new tserver with Master: 73824ea48f6346e682d73d5451b817b8 (127.13.223.1:43843)
I20260806 15:54:39.633553 27058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:60425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:39.651813 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:45197
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:39.773016 27370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:39.773300 27370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:39.773380 27370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:39.777390 27370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:39.777472 27370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:39.777554 27370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:39.782893 27370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:39.784046 27370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:39.785200 27370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:39.795495 27376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:39.795506 27378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:39.795480 27375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:39.795784 27370 server_base.cc:1034] running on GCE node
I20260806 15:54:39.796381 27370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:39.797144 27370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:39.798316 27370 hybrid_clock.cc:648] HybridClock initialized: now 1786031679798269 us; error 60 us; skew 500 ppm
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:39.801416 27370 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:39.802485 27370 webserver.cc:460] Webserver started at http://127.13.223.2:40935/ using document root <none> and password file <none>
I20260806 15:54:39.803097 27370 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:39.803164 27370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:39.803396 27370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:39.805910 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "33873dd765424edfa1038bd1d20432cc"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "0b0fca4e12e649104f369f665395ad61"
server_key_iv: "e19f66a17da82c9cdcd5e02052df2d83"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.806528 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "33873dd765424edfa1038bd1d20432cc"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "0b0fca4e12e649104f369f665395ad61"
server_key_iv: "e19f66a17da82c9cdcd5e02052df2d83"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.810537 27370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:39.813338 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.814677 27370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:39.814779 27370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "33873dd765424edfa1038bd1d20432cc"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "0b0fca4e12e649104f369f665395ad61"
server_key_iv: "e19f66a17da82c9cdcd5e02052df2d83"
server_key_version: "encryptionkey@0"
I20260806 15:54:39.814920 27370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:39.829550 27370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:39.832954 27370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:39.833173 27370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:39.833791 27370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:39.834642 27370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:39.834688 27370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.834757 27370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:39.834796 27370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:39.844926 27370 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:46153
I20260806 15:54:39.844954 27582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:46153 every 8 connection(s)
I20260806 15:54:39.845878 27370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:39.850488 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 27370
I20260806 15:54:39.850617 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:39.850914 14204 external_mini_cluster.cc:1428] Setting key 2125e06438cc633a651cb54c79bf874b
Aug 06 15:54:39 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031679, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:39.860888 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:39.847710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:57845 (local address 127.13.223.62:45197)
0806 15:54:39.847855 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:39.847859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:39.848974 (+ 1115us) server_negotiation.cc:407] Connection header received
0806 15:54:39.850019 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:39.850022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:39.850070 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:39.850148 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:39.851768 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.852286 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.853504 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.853730 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.853836 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:39.856596 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:39.856602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:39.856608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:39.856641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:39.858464 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:39.859004 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:39.859011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:39.859014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:39.859122 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:39.859497 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:39.859500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:39.859502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:39.859760 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:39.859864 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:39.860431 (+ 567us) server_negotiation.cc:299] Negotiation successful
0806 15:54:39.860640 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 15:54:39.861721 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:39.848102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:57845)
0806 15:54:39.848834 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:39.848874 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:39.849670 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:39.850381 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:39.850390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:39.850949 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:39.851611 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:39.851634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.852414 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.852418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:39.853363 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:39.853373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:39.853877 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:39.853881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:39.854059 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:39.854673 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:39.854697 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:39.856403 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:39.858593 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:39.858599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:39.858615 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:39.858866 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:39.859234 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:39.859237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:39.859238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:39.859373 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:39.859863 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:39.859868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:39.860084 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 15:54:39.860338 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 15:54:39.860703 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:54:39.863027 27583 heartbeater.cc:347] Connected to a master server at 127.13.223.62:45197
I20260806 15:54:39.863308 27583 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:39.863868 27583 heartbeater.cc:511] Master 127.13.223.62:45197 requested a full tablet report, sending...
I20260806 15:54:39.864949 27058 ts_manager.cc:194] Registered new tserver with Master: 33873dd765424edfa1038bd1d20432cc (127.13.223.2:46153)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:54:39.865686 27058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:57845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:54:39.881494 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:45197
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:39.994047 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:39.994292 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:39.994334 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:39.998327 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:39.998389 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:39.998447 27594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:40.003793 27594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:40.004935 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:40.006034 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:40.015951 27605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.015933 27602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.015933 27603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.015970 27594 server_base.cc:1034] running on GCE node
I20260806 15:54:40.016664 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:40.017406 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:40.018592 27594 hybrid_clock.cc:648] HybridClock initialized: now 1786031680018565 us; error 36 us; skew 500 ppm
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031680, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:40.021699 27594 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:54:40.022609 27594 webserver.cc:460] Webserver started at http://127.13.223.3:40471/ using document root <none> and password file <none>
I20260806 15:54:40.023155 27594 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:40.023216 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:40.023387 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:40.025542 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "ec9beaaa31934c1085d76ad77a5d19b2"
format_stamp: "Formatted at 2026-08-06 15:54:40 on dist-test-slave-rwrg"
server_key: "98fe051b0944314773c65cc8f65195c7"
server_key_iv: "a4586f837210fceb35971a60684545d1"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.026067 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "ec9beaaa31934c1085d76ad77a5d19b2"
format_stamp: "Formatted at 2026-08-06 15:54:40 on dist-test-slave-rwrg"
server_key: "98fe051b0944314773c65cc8f65195c7"
server_key_iv: "a4586f837210fceb35971a60684545d1"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.029883 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:40.032384 27614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.033738 27594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:40.033897 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "ec9beaaa31934c1085d76ad77a5d19b2"
format_stamp: "Formatted at 2026-08-06 15:54:40 on dist-test-slave-rwrg"
server_key: "98fe051b0944314773c65cc8f65195c7"
server_key_iv: "a4586f837210fceb35971a60684545d1"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.034015 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:40.049765 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:40.053321 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:40.053556 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:40.054201 27594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:40.055073 27594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:40.055153 27594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.055228 27594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:40.055282 27594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.065253 27594 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40671
I20260806 15:54:40.065281 27784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40671 every 8 connection(s)
I20260806 15:54:40.066164 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:40.068363 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 27594
I20260806 15:54:40.068499 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:40.068876 14204 external_mini_cluster.cc:1428] Setting key b2d42f31236e1b6d59ec76e2dc7bbfed
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031680, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:40.080173 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.067930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:35067 (local address 127.13.223.62:45197)
0806 15:54:40.068098 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.068101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.069400 (+ 1299us) server_negotiation.cc:407] Connection header received
0806 15:54:40.070531 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.070534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.070577 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.070667 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.071935 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.072547 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.073671 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.073902 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.073990 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.076540 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:40.076546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:40.076549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:40.076583 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:40.078163 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:40.078782 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:40.078785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:40.078786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:40.078866 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:40.079215 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:40.079217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:40.079218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:40.079385 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:40.079457 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:40.079953 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 15:54:40.080048 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 15:54:40.081107 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.068344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:35067)
0806 15:54:40.069212 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.069275 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.070295 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.070841 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.070850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.071315 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.071791 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.071803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.072681 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.072684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.073563 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.073572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.074040 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.074043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.074182 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.074735 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.074752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.076337 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:40.078295 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:40.078310 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:40.078323 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:40.078674 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:40.078955 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:40.078958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:40.078959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:40.079120 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:40.079486 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:40.079491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:40.079742 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 15:54:40.079913 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 15:54:40.080193 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":177,"threads_started":1}
I20260806 15:54:40.082432 27785 heartbeater.cc:347] Connected to a master server at 127.13.223.62:45197
I20260806 15:54:40.082765 27785 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:40.083323 27785 heartbeater.cc:511] Master 127.13.223.62:45197 requested a full tablet report, sending...
I20260806 15:54:40.084434 27060 ts_manager.cc:194] Registered new tserver with Master: ec9beaaa31934c1085d76ad77a5d19b2 (127.13.223.3:40671)
I20260806 15:54:40.085119 27060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:35067
I20260806 15:54:40.094843 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab.
Entry for principal oryx/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab.
WARNING: no policy specified for oryx/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab.
Entry for principal oryx/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab.
WARNING: no policy specified for oryx/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab.
Entry for principal oryx/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab.
I20260806 15:54:40.174957 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 27154
I20260806 15:54:40.202566 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 27370
I20260806 15:54:40.215375 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 27594
I20260806 15:54:40.225234 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 27009
I20260806 15:54:40.238212 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:45197
--webserver_interface=127.13.223.62
--webserver_port=39457
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:40.351644 27802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:40.351929 27802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:40.351971 27802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:40.356145 27802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:40.356215 27802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:40.356235 27802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:40.356251 27802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:40.356315 27802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:40.361951 27802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=39457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:40.363072 27802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:40.364212 27802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:40.372946 27811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.372937 27809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.372936 27813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.373313 27802 server_base.cc:1034] running on GCE node
I20260806 15:54:40.374003 27802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:40.375180 27802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:40.376410 27802 hybrid_clock.cc:648] HybridClock initialized: now 1786031680376389 us; error 32 us; skew 500 ppm
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031680, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:40.379617 27802 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:40.380707 27802 webserver.cc:460] Webserver started at http://127.13.223.62:39457/ using document root <none> and password file <none>
I20260806 15:54:40.381287 27802 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:40.381405 27802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:40.385408 27802 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 15:54:40.387698 27825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.389098 27802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:40.389207 27802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "9964a6affa69490f9433fb8511d79686"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "dd696f1b63a62519d5868268bf2b9676"
server_key_iv: "4ce6b6c259603c852eb55d8be6ec4b5e"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.389746 27802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:40.402035 27802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:40.405097 27802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:40.405272 27802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:40.412711 27802 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:45197
I20260806 15:54:40.412741 27907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:45197 every 8 connection(s)
I20260806 15:54:40.413822 27802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:40.415194 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 27802
I20260806 15:54:40.415704 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:43843
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:45197
--webserver_port=36555
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:54:40.419090 27908 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:54:40.420948 27908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Bootstrap starting.
I20260806 15:54:40.438455 27908 log.cc:824] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:40.445016 27908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:54:40.445410 27908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Bootstrap complete.
I20260806 15:54:40.449755 27908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:40.450147 27908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964a6affa69490f9433fb8511d79686, State: Initialized, Role: FOLLOWER
I20260806 15:54:40.450718 27908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:40.450838 27908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:40.450927 27908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:40.451025 27908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 1 FOLLOWER]: Advancing to term 2
I20260806 15:54:40.453035 27908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:40.453372 27908 leader_election.cc:302] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9964a6affa69490f9433fb8511d79686; no voters:
I20260806 15:54:40.453719 27908 leader_election.cc:288] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 15:54:40.453809 27913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 15:54:40.454001 27913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [term 2 LEADER]: Becoming Leader. State: Replica: 9964a6affa69490f9433fb8511d79686, State: Running, Role: LEADER
I20260806 15:54:40.454319 27913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } }
I20260806 15:54:40.454985 27908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:40.455339 27918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9964a6affa69490f9433fb8511d79686. Latest consensus state: current_term: 2 leader_uuid: "9964a6affa69490f9433fb8511d79686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } } }
I20260806 15:54:40.455471 27918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:40.455696 27914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9964a6affa69490f9433fb8511d79686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9964a6affa69490f9433fb8511d79686" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 45197 } } }
I20260806 15:54:40.455796 27914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:40.455857 27921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:40.459436 27921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:40.460340 27921 catalog_manager.cc:1259] Loaded cluster ID: ad6c57f2053f4428997ed108515b7b58
I20260806 15:54:40.460400 27921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:40.462980 27921 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:40.463655 27921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9964a6affa69490f9433fb8511d79686: Loaded TSK: 0
I20260806 15:54:40.464956 27921 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 15:54:40.533586 27910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:40.533874 27910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:40.533954 27910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:40.538007 27910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:40.538098 27910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:40.538187 27910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:40.543342 27910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.1.keytab
--principal=oryx/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:43843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=36555
--tserver_master_addrs=127.13.223.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:40.544349 27910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:40.545385 27910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:40.556139 27949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.556207 27951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.556147 27946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.556394 27910 server_base.cc:1034] running on GCE node
I20260806 15:54:40.557010 27910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:40.557682 27910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:40.558890 27910 hybrid_clock.cc:648] HybridClock initialized: now 1786031680558861 us; error 56 us; skew 500 ppm
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031680, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:40.562170 27910 init.cc:377] Logged in from keytab as oryx/127.13.223.1@KRBTEST.COM (short username oryx)
I20260806 15:54:40.563146 27910 webserver.cc:460] Webserver started at http://127.13.223.1:36555/ using document root <none> and password file <none>
I20260806 15:54:40.563760 27910 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:40.563871 27910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:40.567909 27910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:40.570330 27965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.571611 27910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:54:40.571749 27910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "73824ea48f6346e682d73d5451b817b8"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "f996e391c54aff822b3a23db9e146a76"
server_key_iv: "127829d4e867236f36f361c453926138"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.572268 27910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:40.590637 27910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:40.594198 27910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:40.594401 27910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:40.595005 27910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:40.595937 27910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:40.596007 27910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.596097 27910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:40.596148 27910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.606073 27910 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:43843
I20260806 15:54:40.606132 28134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:43843 every 8 connection(s)
I20260806 15:54:40.607113 27910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:40.613340 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 27910
I20260806 15:54:40.613790 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:46153
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:45197
--webserver_port=40935
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:40.621332 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.609457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:39027)
0806 15:54:40.610144 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.610197 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.610981 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.612054 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.612061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.612428 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.612968 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.612984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.614460 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.614467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.615846 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.615857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.616991 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.616993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.617136 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.617777 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.617794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.620483 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":128,"threads_started":1}
W20260806 15:54:40.621891 28135 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:45197 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.223.62:45197: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
W20260806 15:54:40.621968 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.609307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:39027 (local address 127.13.223.62:45197)
0806 15:54:40.609750 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.609772 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.610318 (+ 546us) server_negotiation.cc:407] Connection header received
0806 15:54:40.611330 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.611341 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.611633 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.611935 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.613169 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.614310 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.615983 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.616824 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.617017 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.620849 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:39027: BlockingRecv error: recv got EOF from 127.13.223.1:39027 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:40.628640 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.622681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:53911)
0806 15:54:40.622807 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.622823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.622905 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.623212 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.623215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.623351 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.623442 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.623447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.624606 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.624609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.625134 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.625142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.625786 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.625789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.625856 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.626404 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.626415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.628428 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 15:54:40.628724 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.622791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:53911 (local address 127.13.223.62:45197)
0806 15:54:40.622936 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.622942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.622961 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:54:40.623011 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.623017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.623075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.623192 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.623563 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.624451 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.625260 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.625666 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.625793 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.628613 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:53911: BlockingRecv error: recv got EOF from 127.13.223.1:53911 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:40.634829 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.629556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:49333)
0806 15:54:40.629664 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.629678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.629754 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.629990 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.630024 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.630254 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.630367 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.630374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.631378 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.631382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.631890 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.631897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.632352 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.632356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.632433 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.632964 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.632975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.634691 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 15:54:40.634936 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.629639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:49333 (local address 127.13.223.62:45197)
0806 15:54:40.629767 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.629772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.629803 (+ 31us) server_negotiation.cc:407] Connection header received
0806 15:54:40.629841 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.629849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.629883 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.629975 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.630498 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.631271 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.632011 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.632240 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.632349 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.634803 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:49333: BlockingRecv error: recv got EOF from 127.13.223.1:49333 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 15:54:40.635083 28135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 15:54:40.733767 28142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:40.734036 28142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:40.734078 28142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:40.738101 28142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:40.738161 28142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:40.738257 28142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:40.743495 28142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.2.keytab
--principal=oryx/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:46153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=40935
--tserver_master_addrs=127.13.223.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:40.744567 28142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:40.745589 28142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:40.756124 28152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.756196 28142 server_base.cc:1034] running on GCE node
W20260806 15:54:40.756206 28155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.756445 28158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.756852 28142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:40.757583 28142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:40.758770 28142 hybrid_clock.cc:648] HybridClock initialized: now 1786031680758739 us; error 45 us; skew 500 ppm
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031680, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:40.761787 28142 init.cc:377] Logged in from keytab as oryx/127.13.223.2@KRBTEST.COM (short username oryx)
I20260806 15:54:40.762633 28142 webserver.cc:460] Webserver started at http://127.13.223.2:40935/ using document root <none> and password file <none>
I20260806 15:54:40.763149 28142 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:40.763219 28142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:40.767017 28142 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:40.769117 28171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.770370 28142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:40.770464 28142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "33873dd765424edfa1038bd1d20432cc"
format_stamp: "Formatted at 2026-08-06 15:54:39 on dist-test-slave-rwrg"
server_key: "0b0fca4e12e649104f369f665395ad61"
server_key_iv: "e19f66a17da82c9cdcd5e02052df2d83"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.770977 28142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:40.791179 28142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:40.794800 28142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:40.794963 28142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:40.795526 28142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:40.796326 28142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:40.796372 28142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.796411 28142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:40.796427 28142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.806006 28142 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:46153
I20260806 15:54:40.806035 28351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:46153 every 8 connection(s)
I20260806 15:54:40.807104 28142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:40.811978 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 28142
I20260806 15:54:40.812413 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:40671
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:45197
--webserver_port=40471
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 15:54:40.819639 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.809000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:46461 (local address 127.13.223.62:45197)
0806 15:54:40.809146 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.809150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.810263 (+ 1113us) server_negotiation.cc:407] Connection header received
0806 15:54:40.811188 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.811191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.811256 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.811369 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.812773 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.813315 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.814857 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.815086 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.815200 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.819491 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:46461: BlockingRecv error: recv got EOF from 127.13.223.2:46461 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:40.820271 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.809410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:46461)
0806 15:54:40.810102 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.810146 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.810966 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.811589 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.811597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.811956 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.812546 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.812570 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.813454 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.813458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.814705 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.814720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.815294 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.815300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.815536 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.816188 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.816214 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.819349 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":119,"threads_started":1}
W20260806 15:54:40.820955 28352 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:45197 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.223.62:45197: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:40.827069 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.821447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:57539)
0806 15:54:40.821701 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.821715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.821783 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.822070 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.822073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.822225 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.822323 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.822328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.823472 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.823475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.824019 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.824028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.824724 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.824727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.824801 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.825310 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.825321 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.826981 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260806 15:54:40.827453 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.821579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:57539 (local address 127.13.223.62:45197)
0806 15:54:40.821733 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.821740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.821759 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:54:40.821870 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.821875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.821933 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.822031 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.822462 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.823339 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.824150 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.824600 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.824733 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.827139 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:57539: BlockingRecv error: recv got EOF from 127.13.223.2:57539 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:40.832753 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.827584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:44685)
0806 15:54:40.827769 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.827785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.827912 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.828175 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:40.828177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:40.828299 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:40.828367 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.828371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.829372 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.829374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.829888 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:40.829895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.830431 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.830434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:40.830487 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.830995 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:40.831006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:40.832668 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260806 15:54:40.832947 28352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 15:54:40.833128 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.827668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:44685 (local address 127.13.223.62:45197)
0806 15:54:40.827770 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.827774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.827841 (+ 67us) server_negotiation.cc:407] Connection header received
0806 15:54:40.828006 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.828009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.828053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.828134 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:40.828467 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.829264 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.829994 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:40.830330 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:40.830429 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:40.832810 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:44685: BlockingRecv error: recv got EOF from 127.13.223.2:44685 (error 108)
Metrics: {"server-negotiator.queue_time_us":45,"thread_start_us":127,"threads_started":1}
W20260806 15:54:40.930034 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:40.930294 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:40.930336 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:40.934234 28355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:40.934296 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:40.934353 28355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:40.939505 28355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/krb5kdc/oryx_127.13.223.3.keytab
--principal=oryx/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:40671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=40471
--tserver_master_addrs=127.13.223.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:40.940589 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:40.941633 28355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:40.951455 28366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:40.953222 28367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.953670 28355 server_base.cc:1034] running on GCE node
W20260806 15:54:40.953723 28371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:40.954118 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:40.954807 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:40.955976 28355 hybrid_clock.cc:648] HybridClock initialized: now 1786031680955943 us; error 45 us; skew 500 ppm
Aug 06 15:54:40 dist-test-slave-rwrg krb5kdc[26994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031680, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:40.958844 28355 init.cc:377] Logged in from keytab as oryx/127.13.223.3@KRBTEST.COM (short username oryx)
I20260806 15:54:40.959681 28355 webserver.cc:460] Webserver started at http://127.13.223.3:40471/ using document root <none> and password file <none>
I20260806 15:54:40.960249 28355 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:40.960321 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:40.964466 28355 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260806 15:54:40.966810 28381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.968122 28355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:54:40.968246 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "ec9beaaa31934c1085d76ad77a5d19b2"
format_stamp: "Formatted at 2026-08-06 15:54:40 on dist-test-slave-rwrg"
server_key: "98fe051b0944314773c65cc8f65195c7"
server_key_iv: "a4586f837210fceb35971a60684545d1"
server_key_version: "encryptionkey@0"
I20260806 15:54:40.968875 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:40.979691 28355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:40.982975 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:40.983187 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:40.983795 28355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:40.984689 28355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:40.984737 28355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.984805 28355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:40.984846 28355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:40.994217 28355 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40671
I20260806 15:54:40.994299 28546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40671 every 8 connection(s)
I20260806 15:54:40.995236 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:41.000387 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 28355
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 15:54:41.007315 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:40.997224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:57615 (local address 127.13.223.62:45197)
0806 15:54:40.997368 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:40.997372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:40.998828 (+ 1456us) server_negotiation.cc:407] Connection header received
0806 15:54:40.999701 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:40.999704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:40.999750 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:40.999830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:41.001211 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.001851 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.003005 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.003237 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.003352 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.007162 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:57615: BlockingRecv error: recv got EOF from 127.13.223.3:57615 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:41.008054 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:40.997688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:57615)
0806 15:54:40.998642 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:40.998695 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:40.999485 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:40.999999 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:41.000006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:41.000360 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:41.001018 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.001041 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.001992 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.001995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.002870 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.002880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.003393 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.003397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.003561 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.004174 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:41.004193 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:41.006997 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":602,"threads_started":1}
W20260806 15:54:41.008674 28547 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:45197 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.223.62:45197: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
I20260806 15:54:41.010655 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:41.001436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.13.223.62:45197)
0806 15:54:41.001582 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:41.001587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:41.001613 (+ 26us) server_negotiation.cc:407] Connection header received
0806 15:54:41.001655 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:41.001658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:41.001707 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:41.001803 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:41.002550 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.003068 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.004029 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.004376 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.004510 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.006059 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:41.006068 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:41.006088 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:41.006118 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:41.008314 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:41.008897 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:41.008905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:41.008915 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:41.009106 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:41.009448 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:41.009453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:41.009455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:41.009806 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:41.010023 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:41.010306 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 15:54:41.010539 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:41.015252 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:41.009168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:54683)
0806 15:54:41.009371 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:41.009386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:41.009466 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:41.009706 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:41.009708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:41.009869 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:41.009955 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.009960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.011512 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.011516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.012332 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.012341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.012843 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.012847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.012940 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.013400 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:41.013413 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:41.015110 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260806 15:54:41.015345 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:41.009240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:54683 (local address 127.13.223.62:45197)
0806 15:54:41.009374 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:41.009380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:41.009426 (+ 46us) server_negotiation.cc:407] Connection header received
0806 15:54:41.009551 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:41.009554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:41.009592 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:41.009689 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:41.010066 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.010656 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.012454 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.012730 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.012841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.015228 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:54683: BlockingRecv error: recv got EOF from 127.13.223.3:54683 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:41.020377 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:41.015827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:59179)
0806 15:54:41.015945 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:41.015960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:41.016040 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:41.016252 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:41.016255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:41.016406 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:41.016473 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.016478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.017219 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.017222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.017734 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.017741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.018179 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.018183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.018262 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.018689 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:41.018700 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:41.020291 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 15:54:41.020574 28138 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:41.015918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:59179 (local address 127.13.223.62:45197)
0806 15:54:41.016049 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:41.016053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:41.016067 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:41.016108 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:41.016112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:41.016156 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:41.016236 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:41.016614 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.017085 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.017849 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.018057 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.018177 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.020401 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:59179: BlockingRecv error: recv got EOF from 127.13.223.3:59179 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 15:54:41.020598 28547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:41.642198 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:41.635871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:54111)
0806 15:54:41.636232 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:41.636249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:41.636351 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:41.636652 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:41.636654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:41.636852 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:41.636981 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.636987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.638010 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.638018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.638979 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.638998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.639456 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.639459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.639579 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.640109 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:41.640123 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:41.642078 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
W20260806 15:54:41.642402 28553 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:41.635960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:54111 (local address 127.13.223.62:45197)
0806 15:54:41.636300 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:41.636303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:41.636319 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:41.636444 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:41.636446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:41.636511 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:41.636626 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:41.637146 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.637874 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.639105 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.639338 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.639429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.642225 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:54111: BlockingRecv error: recv got EOF from 127.13.223.1:54111 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:41 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:41.839623 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:41.833737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:38447)
0806 15:54:41.834063 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:41.834079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:41.834194 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:41.834468 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:41.834470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:41.834673 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:41.834774 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.834780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.835560 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.835563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.836172 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:41.836182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.836731 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.836737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:41.836829 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.837446 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:41.837468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:41.839480 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
W20260806 15:54:41.839859 28553 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:41.833858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:38447 (local address 127.13.223.62:45197)
0806 15:54:41.834029 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:41.834047 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:41.834158 (+ 111us) server_negotiation.cc:407] Connection header received
0806 15:54:41.834277 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:41.834279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:41.834334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:41.834414 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:41.834888 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.835432 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.836289 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:41.836607 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:41.836721 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:41.839714 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:38447: BlockingRecv error: recv got EOF from 127.13.223.2:38447 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 15:54:42 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:42 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:42.028689 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:42.021334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:53841)
0806 15:54:42.021753 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:42.021769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:42.021884 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:42.022193 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:42.022196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:42.022393 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:42.022588 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:42.022596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.024013 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.024016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:42.024990 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:42.025002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.025723 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.025730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:42.025837 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:42.026379 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:42.026392 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:42.028460 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":100,"threads_started":1}
W20260806 15:54:42.028824 28553 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:42.021412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:53841 (local address 127.13.223.62:45197)
0806 15:54:42.021637 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:42.021640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:42.021855 (+ 215us) server_negotiation.cc:407] Connection header received
0806 15:54:42.021987 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:42.021989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:42.022044 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:42.022141 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:42.022719 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.023836 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.025150 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.025494 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.025696 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:42.028697 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:53841: BlockingRecv error: recv got EOF from 127.13.223.3:53841 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 15:54:42 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:42 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:42.653393 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:42.643223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:49685)
0806 15:54:42.643545 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:42.643564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:42.643677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:42.644000 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:42.644003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:42.644270 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:42.644427 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:42.644437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.645241 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.645244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:42.645805 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:42.645813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.646278 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.646280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:42.646341 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:42.651215 (+ 4874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:42.651230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:42.653238 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":99,"threads_started":1}
W20260806 15:54:42.653620 28559 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:42.643312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:49685 (local address 127.13.223.62:45197)
0806 15:54:42.643599 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:42.643606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:42.643626 (+ 20us) server_negotiation.cc:407] Connection header received
0806 15:54:42.643793 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:42.643795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:42.643842 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:42.643962 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:42.644584 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.645138 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.645926 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.646173 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.646257 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:42.653462 (+ 7205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:49685: BlockingRecv error: recv got EOF from 127.13.223.1:49685 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
Aug 06 15:54:42 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:42 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:42.847074 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:42.840462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:36967)
0806 15:54:42.840803 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:42.840820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:42.840971 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:42.841256 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:42.841259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:42.841474 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:42.841570 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:42.841576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.842399 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.842405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:42.843394 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:42.843405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.844038 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.844041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:42.844107 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:42.844705 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:42.844722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:42.846869 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
W20260806 15:54:42.847122 28559 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:42.840633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36967 (local address 127.13.223.62:45197)
0806 15:54:42.840800 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:42.840804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:42.840908 (+ 104us) server_negotiation.cc:407] Connection header received
0806 15:54:42.841075 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:42.841078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:42.841141 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:42.841231 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:42.841702 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.842282 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.843532 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:42.843915 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:42.844031 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:42.846994 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:36967: BlockingRecv error: recv got EOF from 127.13.223.2:36967 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 15:54:43 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:43 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:43.035537 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:43.029689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:60245)
0806 15:54:43.030129 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:43.030143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:43.030250 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:43.030506 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:43.030508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:43.030720 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:43.030819 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:43.030825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.031546 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.031549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:43.032114 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:43.032122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.032758 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.032764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:43.032853 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:43.033499 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:43.033521 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:43.035415 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":121,"threads_started":1}
W20260806 15:54:43.035732 28559 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:43.029940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:60245 (local address 127.13.223.62:45197)
0806 15:54:43.030087 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:43.030091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:43.030215 (+ 124us) server_negotiation.cc:407] Connection header received
0806 15:54:43.030333 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:43.030335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:43.030385 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:43.030463 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:43.030927 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.031444 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.032233 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.032588 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.032740 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:43.035557 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:60245: BlockingRecv error: recv got EOF from 127.13.223.3:60245 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 15:54:43 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:43 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:43.660451 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:43.654355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:58271)
0806 15:54:43.654645 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:43.654660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:43.654762 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:43.655103 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:43.655106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:43.655320 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:43.655424 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:43.655430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.656430 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.656435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:43.657201 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:43.657211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.657657 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.657660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:43.657728 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:43.658354 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:43.658367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:43.660261 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
W20260806 15:54:43.660538 28563 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:43.654435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:58271 (local address 127.13.223.62:45197)
0806 15:54:43.654741 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:43.654745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:43.654761 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:43.654834 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:43.654836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:43.654928 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:43.655036 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:43.655546 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.656309 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.657310 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.657525 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.657612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:43.660385 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:58271: BlockingRecv error: recv got EOF from 127.13.223.1:58271 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":156,"threads_started":1}
Aug 06 15:54:43 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:43 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:43.853902 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:43.847937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:42477)
0806 15:54:43.848236 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:43.848250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:43.848371 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:43.848632 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:43.848634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:43.848838 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:43.848938 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:43.848943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.849713 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.849716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:43.850284 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:43.850292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.850794 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.850796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:43.850859 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:43.851513 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:43.851527 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:43.853693 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
W20260806 15:54:43.853973 28563 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:43.848043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:42477 (local address 127.13.223.62:45197)
0806 15:54:43.848248 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:43.848251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:43.848326 (+ 75us) server_negotiation.cc:407] Connection header received
0806 15:54:43.848447 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:43.848449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:43.848512 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:43.848613 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:43.849037 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.849603 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.850389 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:43.850683 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:43.850769 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:43.853852 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:42477: BlockingRecv error: recv got EOF from 127.13.223.2:42477 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 06 15:54:44 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:44 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:44.042570 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:44.036474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:44693)
0806 15:54:44.036867 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:44.036882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:44.036983 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:44.037322 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:44.037325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:44.037586 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:44.037731 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:44.037740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.038629 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.038632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:44.039266 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:44.039274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.039687 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.039690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:44.039760 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:44.040274 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:44.040287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:44.042399 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":108,"threads_started":1}
W20260806 15:54:44.042663 28563 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:44.036747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:44693 (local address 127.13.223.62:45197)
0806 15:54:44.036913 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:44.036919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:44.036939 (+ 20us) server_negotiation.cc:407] Connection header received
0806 15:54:44.037112 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:44.037121 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:44.037175 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:44.037289 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:44.037846 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.038511 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.039372 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.039588 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.039663 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:44.042569 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:44693: BlockingRecv error: recv got EOF from 127.13.223.3:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 15:54:44 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:44 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:44.666848 28569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:44.661364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:58829)
0806 15:54:44.661661 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:44.661680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:44.661775 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:44.662006 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:44.662026 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:44.662268 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:44.662386 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:44.662393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.663167 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.663169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:44.663709 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:44.663716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.664158 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.664160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:44.664221 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:44.664835 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:44.664847 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:44.666730 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
W20260806 15:54:44.667070 28572 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:44.661502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:58829 (local address 127.13.223.62:45197)
0806 15:54:44.661777 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:44.661781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:44.661797 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:54:44.661838 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:44.661840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:44.661885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:44.661973 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:44.662502 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.663063 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.663817 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.664054 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.664142 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:44.666873 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:58829: BlockingRecv error: recv got EOF from 127.13.223.1:58829 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
Aug 06 15:54:44 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:44 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:44.861124 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:44.854790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:42491)
0806 15:54:44.855182 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:44.855197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:44.855299 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:44.855610 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:44.855613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:44.855797 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:44.855937 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:44.855947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.857042 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.857046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:44.857858 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:44.857868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.858347 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.858352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:44.858434 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:44.859030 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:44.859043 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:44.860990 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":113,"threads_started":1}
W20260806 15:54:44.861330 28572 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:44.854926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:42491 (local address 127.13.223.62:45197)
0806 15:54:44.855172 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:44.855175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:44.855272 (+ 97us) server_negotiation.cc:407] Connection header received
0806 15:54:44.855384 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:44.855387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:44.855446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:44.855581 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:44.856109 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.856880 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.857985 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:44.858224 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:44.858340 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:44.861202 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:42491: BlockingRecv error: recv got EOF from 127.13.223.2:42491 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 15:54:45 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:45 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:45.049618 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:45.043577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:51501)
0806 15:54:45.043993 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:45.044013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:45.044187 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:45.044550 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:45.044553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:45.044837 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:45.044988 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:45.045000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.045933 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.045935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:45.046489 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:45.046497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.046919 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.046921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:45.046982 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:45.047561 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:45.047582 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:45.049495 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":120,"threads_started":1}
W20260806 15:54:45.049808 28572 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:45.043688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:51501 (local address 127.13.223.62:45197)
0806 15:54:45.043881 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:45.043887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:45.044151 (+ 264us) server_negotiation.cc:407] Connection header received
0806 15:54:45.044293 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:45.044297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:45.044368 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:45.044445 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:45.045164 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.045824 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.046592 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.046820 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.046897 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:45.049630 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:51501: BlockingRecv error: recv got EOF from 127.13.223.3:51501 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 15:54:45 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:45 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:45.673303 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:45.667588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:57555)
0806 15:54:45.667876 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:45.667889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:45.667978 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:45.668234 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:45.668236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:45.668418 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:45.668563 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:45.668569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.669416 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.669419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:45.669997 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:45.670006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.670423 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.670426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:45.670486 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:45.671007 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:45.671025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:45.673119 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
W20260806 15:54:45.673432 28579 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:45.667689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:57555 (local address 127.13.223.62:45197)
0806 15:54:45.667926 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:45.667929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:45.667943 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:45.668064 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:45.668066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:45.668122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:45.668201 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:45.668672 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.669311 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.670101 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.670325 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.670401 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:45.673286 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:57555: BlockingRecv error: recv got EOF from 127.13.223.1:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
Aug 06 15:54:45 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:45 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:45.867498 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:45.861995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:39593)
0806 15:54:45.862312 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:45.862327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:45.862431 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:45.862713 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:45.862716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:45.862906 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:45.863002 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:45.863008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.863807 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.863810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:45.864400 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:45.864407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.864892 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.864895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:45.864958 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:45.865487 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:45.865499 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:45.867386 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
W20260806 15:54:45.867659 28579 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:45.862178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:39593 (local address 127.13.223.62:45197)
0806 15:54:45.862340 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:45.862343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:45.862385 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:54:45.862517 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:45.862519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:45.862562 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:45.862642 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:45.863165 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.863702 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.864527 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:45.864769 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:45.864854 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:45.867515 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:39593: BlockingRecv error: recv got EOF from 127.13.223.2:39593 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 15:54:46 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:46 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:46.057360 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:46.050707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:56493)
0806 15:54:46.051013 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:46.051029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:46.051162 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:46.051407 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:46.051409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:46.051600 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:46.051698 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:46.051703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.052672 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.052677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:46.053326 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:46.053335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.053904 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.053908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:46.054010 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:46.054772 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:46.054790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:46.057166 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
W20260806 15:54:46.057418 28579 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:46.050735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:56493 (local address 127.13.223.62:45197)
0806 15:54:46.050978 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:46.050982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:46.051105 (+ 123us) server_negotiation.cc:407] Connection header received
0806 15:54:46.051236 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:46.051239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:46.051287 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:46.051373 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:46.051852 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.052500 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.053468 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.053766 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.053894 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:46.057300 (+ 3406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:56493: BlockingRecv error: recv got EOF from 127.13.223.3:56493 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Aug 06 15:54:46 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:46 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:46.680186 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:46.674307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:50327)
0806 15:54:46.674722 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:46.674737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:46.674854 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:46.675140 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:46.675142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:46.675386 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:46.675491 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:46.675497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.676362 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.676365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:46.677034 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:46.677044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.677485 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.677487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:46.677596 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:46.678120 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:46.678132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:46.680009 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":114,"threads_started":1}
W20260806 15:54:46.680347 28583 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:46.674383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:50327 (local address 127.13.223.62:45197)
0806 15:54:46.674737 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:46.674748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:46.674800 (+ 52us) server_negotiation.cc:407] Connection header received
0806 15:54:46.674967 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:46.674969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:46.675019 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:46.675101 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:46.675646 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.676264 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.677147 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.677386 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.677465 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:46.680194 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:50327: BlockingRecv error: recv got EOF from 127.13.223.1:50327 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":70,"threads_started":1}
Aug 06 15:54:46 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:46 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:46.874464 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:46.868374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:40209)
0806 15:54:46.868700 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:46.868714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:46.868857 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:46.869233 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:46.869236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:46.869495 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:46.869689 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:46.869699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.870605 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.870608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:46.871152 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:46.871159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.871593 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.871596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:46.871656 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:46.872174 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:46.872187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:46.874300 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
W20260806 15:54:46.874524 28583 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:46.868442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:40209 (local address 127.13.223.62:45197)
0806 15:54:46.868647 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:46.868652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:46.868795 (+ 143us) server_negotiation.cc:407] Connection header received
0806 15:54:46.868994 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:46.868996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:46.869066 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:46.869175 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:46.869810 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.870502 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.871252 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:46.871461 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:46.871536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:46.874420 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:40209: BlockingRecv error: recv got EOF from 127.13.223.2:40209 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 15:54:47 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:47 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:47.064009 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:47.058439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:42511)
0806 15:54:47.058815 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:47.058831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:47.058941 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:47.059205 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:47.059208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:47.059408 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:47.059509 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:47.059515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.060305 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.060308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:47.060981 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:47.060991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.061409 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.061412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:47.061472 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:47.062001 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:47.062013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:47.063884 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
W20260806 15:54:47.064268 28583 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:47.058535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:42511 (local address 127.13.223.62:45197)
0806 15:54:47.058685 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:47.058688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:47.058911 (+ 223us) server_negotiation.cc:407] Connection header received
0806 15:54:47.059010 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:47.059012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:47.059057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:47.059128 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:47.059674 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.060207 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.061088 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.061310 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.061392 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:47.064121 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:42511: BlockingRecv error: recv got EOF from 127.13.223.3:42511 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 06 15:54:47 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:47 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:47.691093 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:47.681152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:46475)
0806 15:54:47.681508 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:47.681523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:47.681629 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:47.681884 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:47.681886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:47.682068 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:47.682174 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:47.682180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.682955 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.682957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:47.683523 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:47.683531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.683946 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.683948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:47.684018 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:47.688717 (+ 4699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:47.688729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:47.690911 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":78,"threads_started":1}
W20260806 15:54:47.691188 28591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:47.681223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:46475 (local address 127.13.223.62:45197)
0806 15:54:47.681526 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:47.681529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:47.681599 (+ 70us) server_negotiation.cc:407] Connection header received
0806 15:54:47.681712 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:47.681714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:47.681758 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:47.681845 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:47.682309 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.682855 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.683627 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.683843 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.683923 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:47.691036 (+ 7113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:46475: BlockingRecv error: recv got EOF from 127.13.223.1:46475 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
Aug 06 15:54:47 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:47 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:47.882068 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:47.875578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:52467)
0806 15:54:47.875923 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:47.875937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:47.876054 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:47.876333 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:47.876336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:47.876644 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:47.876761 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:47.876768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.877863 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.877865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:47.878413 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:47.878421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.878970 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.878977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:47.879093 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:47.879777 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:47.879797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:47.881931 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
W20260806 15:54:47.882258 28591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:47.875594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:52467 (local address 127.13.223.62:45197)
0806 15:54:47.875798 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:47.875802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:47.876063 (+ 261us) server_negotiation.cc:407] Connection header received
0806 15:54:47.876135 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:47.876138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:47.876185 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:47.876255 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:47.876926 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.877687 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.878538 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:47.878837 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:47.878945 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:47.882085 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:52467: BlockingRecv error: recv got EOF from 127.13.223.2:52467 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 06 15:54:48 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:48 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:48.071255 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:48.064990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:51317)
0806 15:54:48.065436 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:48.065452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:48.065615 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:48.065961 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:48.065965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:48.066209 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:48.066310 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:48.066316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.067177 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.067185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:48.067757 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:48.067766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.068265 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.068269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:48.068363 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:48.068960 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:48.068973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:48.071070 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":181,"threads_started":1}
W20260806 15:54:48.071311 28591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:48.065159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:51317 (local address 127.13.223.62:45197)
0806 15:54:48.065358 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:48.065364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:48.065550 (+ 186us) server_negotiation.cc:407] Connection header received
0806 15:54:48.065723 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:48.065727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:48.065800 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:48.065926 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:48.066442 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.067049 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.067891 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.068142 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.068263 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:48.071198 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:51317: BlockingRecv error: recv got EOF from 127.13.223.3:51317 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 15:54:48 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:48 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:48.698011 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:48.692013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:40397)
0806 15:54:48.692321 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:48.692334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:48.692469 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:48.692871 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:48.692876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:48.693128 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:48.693249 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:48.693257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.694018 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.694020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:48.694729 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:48.694738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.695176 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.695178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:48.695248 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:48.695734 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:48.695745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:48.697833 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
W20260806 15:54:48.698096 28596 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:48.692188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:40397 (local address 127.13.223.62:45197)
0806 15:54:48.692407 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:48.692413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:48.692430 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:54:48.692657 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:48.692661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:48.692717 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:48.692814 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:48.693374 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.693923 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.694838 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.695075 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.695158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:48.697967 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:40397: BlockingRecv error: recv got EOF from 127.13.223.1:40397 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
Aug 06 15:54:48 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:48 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:48.888708 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:48.882873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:51613)
0806 15:54:48.883181 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:48.883199 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:48.883342 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:48.883632 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:48.883634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:48.883855 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:48.883949 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:48.883955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.884914 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.884919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:48.885483 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:48.885491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.885967 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.885970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:48.886053 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:48.886596 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:48.886609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:48.888500 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20260806 15:54:48.888765 28596 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:48.882944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:51613 (local address 127.13.223.62:45197)
0806 15:54:48.883139 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:48.883144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:48.883290 (+ 146us) server_negotiation.cc:407] Connection header received
0806 15:54:48.883455 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:48.883459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:48.883511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:48.883608 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:48.884081 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.884805 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.885619 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:48.885847 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:48.885961 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:48.888655 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:51613: BlockingRecv error: recv got EOF from 127.13.223.2:51613 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 15:54:49 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:49 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:49.078204 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:49.072166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:41189)
0806 15:54:49.072528 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:49.072551 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:49.072669 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:49.072999 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:49.073001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:49.073200 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:49.073300 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:49.073306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.074158 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.074161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:49.074825 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:49.074835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.075285 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.075288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:49.075364 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:49.075983 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:49.075995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:49.078018 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
W20260806 15:54:49.078351 28596 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:49.072222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:41189 (local address 127.13.223.62:45197)
0806 15:54:49.072372 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:49.072376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:49.072661 (+ 285us) server_negotiation.cc:407] Connection header received
0806 15:54:49.072759 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:49.072761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:49.072848 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:49.072925 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:49.073451 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.074050 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.074939 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.075178 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.075275 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:49.078230 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:41189: BlockingRecv error: recv got EOF from 127.13.223.3:41189 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 15:54:49 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:49 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:49.705019 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:49.698991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:58953)
0806 15:54:49.699297 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:49.699311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:49.699411 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:49.699665 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:49.699668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:49.699878 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:49.699974 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:49.699979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.700795 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.700798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:49.701733 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:49.701745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.702212 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.702215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:49.702273 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:49.702771 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:49.702783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:49.704847 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":119,"threads_started":1}
W20260806 15:54:49.705376 28604 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:49.699196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:58953 (local address 127.13.223.62:45197)
0806 15:54:49.699431 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:49.699434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:49.699447 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:54:49.699483 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:49.699485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:49.699529 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:49.699608 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:49.700124 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.700686 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.701872 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.702112 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.702201 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:49.705154 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:58953: BlockingRecv error: recv got EOF from 127.13.223.1:58953 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
Aug 06 15:54:49 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:49 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:49.895705 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:49.889676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:40925)
0806 15:54:49.889967 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:49.889980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:49.890112 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:49.890405 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:49.890408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:49.890635 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:49.890739 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:49.890745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.891502 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.891507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:49.892104 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:49.892115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.892730 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.892734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:49.892840 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:49.893456 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:49.893470 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:49.895566 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
W20260806 15:54:49.895797 28604 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:49.889687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:40925 (local address 127.13.223.62:45197)
0806 15:54:49.889857 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:49.889863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:49.890057 (+ 194us) server_negotiation.cc:407] Connection header received
0806 15:54:49.890232 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:49.890236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:49.890294 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:49.890383 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:49.890858 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.891384 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.892240 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:49.892598 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:49.892728 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:49.895666 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:40925: BlockingRecv error: recv got EOF from 127.13.223.2:40925 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 15:54:50 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:50 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:50.085671 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:50.079138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:37665)
0806 15:54:50.079521 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:50.079571 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:50.079726 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:50.080018 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:50.080021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:50.080314 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:50.080450 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:50.080458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.081436 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.081440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:50.082229 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:50.082239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.082693 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.082696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:50.082765 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:50.083381 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:50.083396 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:50.085544 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":103,"threads_started":1}
W20260806 15:54:50.085865 28604 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:50.079273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:37665 (local address 127.13.223.62:45197)
0806 15:54:50.079464 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:50.079469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:50.079664 (+ 195us) server_negotiation.cc:407] Connection header received
0806 15:54:50.079846 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:50.079849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:50.079905 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:50.079991 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:50.080613 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.081337 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.082341 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.082561 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.082644 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:50.085726 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:37665: BlockingRecv error: recv got EOF from 127.13.223.3:37665 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 06 15:54:50 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:50 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:50.711447 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:50.706084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:38093)
0806 15:54:50.706358 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:50.706372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:50.706452 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:50.706702 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:50.706704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:50.706887 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:50.706980 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:50.706985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.707777 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.707779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:50.708338 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:50.708345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.708814 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.708816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:50.708879 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:50.709491 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:50.709503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:50.711316 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
W20260806 15:54:50.711635 28611 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:50.706135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:38093 (local address 127.13.223.62:45197)
0806 15:54:50.706398 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:50.706402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:50.706438 (+ 36us) server_negotiation.cc:407] Connection header received
0806 15:54:50.706542 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:50.706545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:50.706589 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:50.706668 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:50.707115 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.707677 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.708440 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.708702 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.708781 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:50.711460 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:38093: BlockingRecv error: recv got EOF from 127.13.223.1:38093 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
Aug 06 15:54:50 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:50 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:50.902172 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:50.896520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:48273)
0806 15:54:50.896856 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:50.896870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:50.896966 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:50.897213 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:50.897215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:50.897397 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:50.897509 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:50.897515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.898232 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.898234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:50.898777 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:50.898784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.899207 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.899210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:50.899270 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:50.899785 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:50.899796 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:50.902041 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
W20260806 15:54:50.902283 28611 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:50.896602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:48273 (local address 127.13.223.62:45197)
0806 15:54:50.896744 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:50.896748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:50.896942 (+ 194us) server_negotiation.cc:407] Connection header received
0806 15:54:50.897046 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:50.897048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:50.897090 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:50.897174 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:50.897608 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.898136 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.898873 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:50.899091 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:50.899176 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:50.902188 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:48273: BlockingRecv error: recv got EOF from 127.13.223.2:48273 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 15:54:51 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:51 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:51.092576 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:51.086552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:52737)
0806 15:54:51.086901 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:51.086918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:51.087053 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:51.087339 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:51.087342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:51.087567 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:51.087664 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:51.087670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.088460 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.088465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:51.089083 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:51.089092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.089566 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.089570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:51.089661 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:51.090189 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:51.090201 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:51.092368 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
W20260806 15:54:51.092717 28611 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:51.086739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:52737 (local address 127.13.223.62:45197)
0806 15:54:51.086897 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:51.086902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:51.086981 (+ 79us) server_negotiation.cc:407] Connection header received
0806 15:54:51.087167 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:51.087170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:51.087224 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:51.087312 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:51.087786 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.088348 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.089209 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.089448 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.089568 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:51.092596 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:52737: BlockingRecv error: recv got EOF from 127.13.223.3:52737 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 15:54:51 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:51 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:51.719447 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:51.712452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:44683)
0806 15:54:51.712814 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:51.712831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:51.712995 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:51.713387 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:51.713390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:51.713606 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:51.713710 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:51.713715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.714701 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.714705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:51.715550 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:51.715563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.716413 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.716421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:51.716542 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:51.717212 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:51.717224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:51.719270 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":112,"threads_started":1}
W20260806 15:54:51.719529 28621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:51.712892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:44683 (local address 127.13.223.62:45197)
0806 15:54:51.713147 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:51.713151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:51.713166 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:54:51.713206 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:51.713208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:51.713250 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:51.713363 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:51.713849 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.714578 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.715714 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.716159 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.716266 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:51.719402 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:44683: BlockingRecv error: recv got EOF from 127.13.223.1:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
Aug 06 15:54:51 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:51 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:51.909490 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:51.903242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:45747)
0806 15:54:51.903519 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:51.903532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:51.903654 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:51.903921 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:51.903922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:51.904155 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:51.904256 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:51.904262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.905316 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.905320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:51.906165 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:51.906174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.906638 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.906641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:51.906701 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:51.907209 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:51.907224 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:51.909315 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
W20260806 15:54:51.909601 28621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:51.903304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:45747 (local address 127.13.223.62:45197)
0806 15:54:51.903460 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:51.903464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:51.903645 (+ 181us) server_negotiation.cc:407] Connection header received
0806 15:54:51.903737 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:51.903739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:51.903798 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:51.903879 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:51.904412 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.905164 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.906280 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:51.906500 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:51.906623 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:51.909441 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:45747: BlockingRecv error: recv got EOF from 127.13.223.2:45747 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 15:54:52 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:52 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:52.099368 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:52.093617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:53177)
0806 15:54:52.093898 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:52.093911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:52.094075 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:52.094392 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:52.094397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:52.094650 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:52.094751 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:52.094758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.095506 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.095509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:52.096124 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:52.096132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.096669 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.096673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:52.096752 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:52.097231 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:52.097243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:52.099175 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
W20260806 15:54:52.099406 28621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:52.093640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:53177 (local address 127.13.223.62:45197)
0806 15:54:52.093793 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:52.093799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:52.094010 (+ 211us) server_negotiation.cc:407] Connection header received
0806 15:54:52.094192 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:52.094199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:52.094248 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:52.094356 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:52.094866 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.095404 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.096240 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.096459 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.096615 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:52.099298 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:53177: BlockingRecv error: recv got EOF from 127.13.223.3:53177 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 15:54:52 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:52 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:52.730134 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:52.720297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:46953)
0806 15:54:52.720620 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:52.720641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:52.720738 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:52.720992 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:52.720994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:52.721191 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:52.721289 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:52.721294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.722087 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.722089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:52.722635 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:52.722642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.723111 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.723114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:52.723172 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:52.727655 (+ 4483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:52.727671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:52.729996 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
W20260806 15:54:52.730314 28627 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:52.720400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:46953 (local address 127.13.223.62:45197)
0806 15:54:52.720735 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:52.720739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:52.720753 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:52.720814 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:52.720816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:52.720865 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:52.720950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:52.721406 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.721986 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.722768 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.722994 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.723084 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:52.730134 (+ 7050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:46953: BlockingRecv error: recv got EOF from 127.13.223.1:46953 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
Aug 06 15:54:52 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:52 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:52.916026 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:52.910336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:43351)
0806 15:54:52.910677 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:52.910698 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:52.910840 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:52.911120 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:52.911123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:52.911339 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:52.911438 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:52.911444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.912240 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.912245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:52.912838 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:52.912847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.913329 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.913333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:52.913419 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:52.913938 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:52.913950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:52.915892 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":86,"threads_started":1}
W20260806 15:54:52.916198 28627 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:52.910480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:43351 (local address 127.13.223.62:45197)
0806 15:54:52.910649 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:52.910653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:52.910785 (+ 132us) server_negotiation.cc:407] Connection header received
0806 15:54:52.910942 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:52.910946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:52.911002 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:52.911098 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:52.911565 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.912123 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.912964 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:52.913203 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:52.913313 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:52.916058 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:43351: BlockingRecv error: recv got EOF from 127.13.223.2:43351 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 15:54:53 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:53 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:53.105831 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:53.100191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:47053)
0806 15:54:53.100557 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:53.100574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:53.100733 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:53.101014 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:53.101016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:53.101247 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:53.101341 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:53.101346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.102151 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.102156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:53.102715 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:53.102723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.103180 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.103184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:53.103270 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:53.103783 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:53.103795 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:53.105712 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":94,"threads_started":1}
W20260806 15:54:53.106037 28627 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:53.100337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:47053 (local address 127.13.223.62:45197)
0806 15:54:53.100541 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:53.100546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:53.100668 (+ 122us) server_negotiation.cc:407] Connection header received
0806 15:54:53.100838 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:53.100841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:53.100895 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:53.100985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:53.101462 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.102032 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.102837 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.103062 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.103174 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:53.105890 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:47053: BlockingRecv error: recv got EOF from 127.13.223.3:47053 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 15:54:53 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:53 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:53.736653 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:53.731033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:38859)
0806 15:54:53.731343 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:53.731362 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:53.731524 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:53.731799 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:53.731801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:53.731984 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:53.732081 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:53.732086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.732956 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.732958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:53.733502 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:53.733510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.733944 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.733946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:53.734006 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:53.734536 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:53.734547 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:53.736462 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
W20260806 15:54:53.736879 28639 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:53.731092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:38859 (local address 127.13.223.62:45197)
0806 15:54:53.731378 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:53.731385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:53.731481 (+ 96us) server_negotiation.cc:407] Connection header received
0806 15:54:53.731634 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:53.731637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:53.731683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:53.731760 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:53.732187 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.732859 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.733611 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.733831 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.733921 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:53.736730 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:38859: BlockingRecv error: recv got EOF from 127.13.223.1:38859 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":65,"threads_started":1}
Aug 06 15:54:53 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:53 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:53.923125 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:53.916888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:40525)
0806 15:54:53.917264 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:53.917282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:53.917435 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:53.917729 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:53.917732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:53.917944 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:53.918037 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:53.918042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.918856 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.918861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:53.919612 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:53.919621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.920229 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.920236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:53.920327 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:53.920926 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:53.920939 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:53.922958 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
W20260806 15:54:53.923215 28639 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:53.917095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:40525 (local address 127.13.223.62:45197)
0806 15:54:53.917267 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:53.917273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:53.917373 (+ 100us) server_negotiation.cc:407] Connection header received
0806 15:54:53.917546 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:53.917549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:53.917615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:53.917702 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:53.918168 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.918741 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.919731 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:53.920009 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:53.920136 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:53.923121 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:40525: BlockingRecv error: recv got EOF from 127.13.223.2:40525 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 15:54:54 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:54 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:54.112776 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:54.106705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:50345)
0806 15:54:54.107114 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:54.107129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:54.107289 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:54.107667 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:54.107670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:54.107943 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:54.108060 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:54.108067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.108939 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.108943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:54.109534 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:54.109542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.110016 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.110020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:54.110118 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:54.110627 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:54.110639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:54.112653 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":98,"threads_started":1}
W20260806 15:54:54.112911 28639 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:54.106862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:50345 (local address 127.13.223.62:45197)
0806 15:54:54.107052 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:54.107057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:54.107225 (+ 168us) server_negotiation.cc:407] Connection header received
0806 15:54:54.107435 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:54.107437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:54.107490 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:54.107646 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:54.108192 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.108829 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.109669 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.109895 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.110002 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:54.112789 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:50345: BlockingRecv error: recv got EOF from 127.13.223.3:50345 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 15:54:54 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:54 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:54.743917 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:54.737473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:46713)
0806 15:54:54.737827 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:54.737845 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:54.737964 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:54.738223 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:54.738225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:54.738409 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:54.738508 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:54.738513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.739461 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.739464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:54.740337 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:54.740351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.740863 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.740866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:54.740936 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:54.741433 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:54.741444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:54.743747 (+ 2303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
W20260806 15:54:54.743996 28645 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:54.737607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:46713 (local address 127.13.223.62:45197)
0806 15:54:54.737963 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:54.737966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:54.737980 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:54:54.738061 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:54.738063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:54.738107 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:54.738184 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:54.738637 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.739303 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.740486 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.740736 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.740822 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:54.743871 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:46713: BlockingRecv error: recv got EOF from 127.13.223.1:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":90,"threads_started":1}
Aug 06 15:54:54 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:54 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:54.930034 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:54.923975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:60723)
0806 15:54:54.924292 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:54.924315 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:54.924457 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:54.924798 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:54.924801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:54.925020 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:54.925138 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:54.925144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.925920 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.925925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:54.926636 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:54.926647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.927164 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.927168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:54.927271 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:54.927798 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:54.927810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:54.929891 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
W20260806 15:54:54.930167 28645 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:54.924088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:60723 (local address 127.13.223.62:45197)
0806 15:54:54.924260 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:54.924264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:54.924399 (+ 135us) server_negotiation.cc:407] Connection header received
0806 15:54:54.924637 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:54.924640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:54.924687 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:54.924780 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:54.925276 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.925809 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.926744 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:54.927005 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:54.927120 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:54.930029 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:60723: BlockingRecv error: recv got EOF from 127.13.223.2:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 15:54:55 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:55 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.3@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:55.119609 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:55.113636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.3:33761)
0806 15:54:55.113978 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:55.113995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:55.114161 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:55.114445 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:55.114448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:55.114669 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:55.114763 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:55.114768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.115537 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.115540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:55.116129 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:55.116138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.116659 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.116663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:55.116753 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:55.117302 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:55.117314 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:55.119414 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
W20260806 15:54:55.119771 28645 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:55.113799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:33761 (local address 127.13.223.62:45197)
0806 15:54:55.113978 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:55.113984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:55.114102 (+ 118us) server_negotiation.cc:407] Connection header received
0806 15:54:55.114271 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:55.114275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:55.114331 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:55.114420 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:55.114888 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.115427 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.116256 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.116485 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.116657 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:55.119642 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.3:33761: BlockingRecv error: recv got EOF from 127.13.223.3:33761 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 15:54:55 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:55 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.1@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:55.751783 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:55.744973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.1:47753)
0806 15:54:55.745447 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:55.745465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:55.745600 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:55.745960 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:55.745963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:55.746247 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:55.746387 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:55.746394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.747327 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.747332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:55.748263 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:55.748274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.748945 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.748948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:55.749045 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:55.749661 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:55.749676 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:55.751603 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":87,"threads_started":1}
W20260806 15:54:55.751864 28649 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:55.745109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:47753 (local address 127.13.223.62:45197)
0806 15:54:55.745384 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:55.745391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:55.745539 (+ 148us) server_negotiation.cc:407] Connection header received
0806 15:54:55.745717 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:55.745742 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:55.745803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:55.745912 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:55.746518 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.747213 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.748402 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.748822 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.748937 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:55.751736 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.1:47753: BlockingRecv error: recv got EOF from 127.13.223.1:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
Aug 06 15:54:55 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 15:54:55 dist-test-slave-rwrg krb5kdc[26994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.2@KRBTEST.COM for oryx/127.13.223.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 15:54:55.937681 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:55.931122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:45197 (local address 127.13.223.2:44015)
0806 15:54:55.931495 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:55.931515 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:55.931681 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:55.931993 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:55.931996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:55.932257 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:55.932361 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:55.932367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.933539 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.933543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:55.934398 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:55.934409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.934937 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.934940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:55.935011 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:55.935551 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:55.935563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:55.937564 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.62:45197: Server oryx/127.13.223.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":109,"threads_started":1}
W20260806 15:54:55.937840 28649 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:55.931280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:44015 (local address 127.13.223.62:45197)
0806 15:54:55.931495 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:55.931502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:55.931618 (+ 116us) server_negotiation.cc:407] Connection header received
0806 15:54:55.931798 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:55.931802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:55.931856 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:55.931965 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:55.932536 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.933339 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.934549 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:55.934821 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:55.934912 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:55.937713 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.2:44015: BlockingRecv error: recv got EOF from 127.13.223.2:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 15:54:56.003139 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 27910
I20260806 15:54:56.017822 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 28142
I20260806 15:54:56.029059 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 28355
I20260806 15:54:56.038968 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 27802
2026-08-06T15:54:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18414 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T15:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:56Z Disabled control of system clock
I20260806 15:54:56.075314 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34921
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 15:54:56.189436 28657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:56.189713 28657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:56.189747 28657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:56.194020 28657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:56.194082 28657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:56.194104 28657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:56.194123 28657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:56.194137 28657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:56.198846 28657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:56.199885 28657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:56.201359 28657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:54:56.210237 28657 server_base.cc:1034] running on GCE node
W20260806 15:54:56.210216 28664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.210287 28669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.210287 28663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:56.211251 28657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:56.212301 28657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:56.213500 28657 hybrid_clock.cc:648] HybridClock initialized: now 1786031696213486 us; error 38 us; skew 500 ppm
I20260806 15:54:56.215315 28657 webserver.cc:460] Webserver started at http://127.13.223.62:42429/ using document root <none> and password file <none>
I20260806 15:54:56.215943 28657 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:56.216038 28657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:56.216266 28657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:56.218392 28657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "de96dfc1631b4c52b95f2c5f2732064c"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "2cd33482c6e44b57dc455145ec8d6c44"
server_key_iv: "cc9f933aa9bbe2664c4ef3cc6d070f5d"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.218955 28657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "de96dfc1631b4c52b95f2c5f2732064c"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "2cd33482c6e44b57dc455145ec8d6c44"
server_key_iv: "cc9f933aa9bbe2664c4ef3cc6d070f5d"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.222716 28657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:54:56.225275 28679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.226593 28657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:56.226742 28657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "de96dfc1631b4c52b95f2c5f2732064c"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "2cd33482c6e44b57dc455145ec8d6c44"
server_key_iv: "cc9f933aa9bbe2664c4ef3cc6d070f5d"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.226866 28657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:56.237381 28657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:56.238044 28657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:56.238246 28657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:56.245702 28657 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34921
I20260806 15:54:56.245726 28763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34921 every 8 connection(s)
I20260806 15:54:56.246691 28657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:56.250010 28765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:56.251753 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 28657
I20260806 15:54:56.251854 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:56.252173 14204 external_mini_cluster.cc:1428] Setting key 06f91ea8ecce617df66f7b6fc6a7466e
I20260806 15:54:56.256686 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c: Bootstrap starting.
I20260806 15:54:56.259294 28765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:56.260043 28765 log.cc:824] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:56.261883 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.253657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.13.223.62:34921)
0806 15:54:56.254262 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:56.254278 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:56.254312 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:54:56.254965 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:56.254982 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:56.255372 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:56.255645 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:56.256073 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.257122 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.258301 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.258737 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.258848 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.259778 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:56.259783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:56.259796 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:56.260185 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:56.260245 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:56.260265 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:56.260424 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:56.260655 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 15:54:56.260975 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260806 15:54:56.262449 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c: No bootstrap required, opened a new log
I20260806 15:54:56.266216 28765 raft_consensus.cc:348] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96dfc1631b4c52b95f2c5f2732064c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34921 } }
I20260806 15:54:56.266434 28765 raft_consensus.cc:374] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:56.266469 28765 raft_consensus.cc:729] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de96dfc1631b4c52b95f2c5f2732064c, State: Initialized, Role: FOLLOWER
I20260806 15:54:56.266881 28765 consensus_queue.cc:260] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96dfc1631b4c52b95f2c5f2732064c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34921 } }
I20260806 15:54:56.267005 28765 raft_consensus.cc:388] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:56.267046 28765 raft_consensus.cc:482] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:56.267112 28765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:56.267968 28765 raft_consensus.cc:504] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96dfc1631b4c52b95f2c5f2732064c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34921 } }
I20260806 15:54:56.268251 28765 leader_election.cc:302] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de96dfc1631b4c52b95f2c5f2732064c; no voters:
I20260806 15:54:56.268576 28765 leader_election.cc:288] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:56.268731 28773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:56.269001 28773 raft_consensus.cc:686] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [term 1 LEADER]: Becoming Leader. State: Replica: de96dfc1631b4c52b95f2c5f2732064c, State: Running, Role: LEADER
I20260806 15:54:56.269423 28773 consensus_queue.cc:237] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96dfc1631b4c52b95f2c5f2732064c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34921 } }
I20260806 15:54:56.269848 28765 sys_catalog.cc:564] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:56.271157 28779 sys_catalog.cc:455] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [sys.catalog]: SysCatalogTable state changed. Reason: New leader de96dfc1631b4c52b95f2c5f2732064c. Latest consensus state: current_term: 1 leader_uuid: "de96dfc1631b4c52b95f2c5f2732064c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96dfc1631b4c52b95f2c5f2732064c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34921 } } }
I20260806 15:54:56.271292 28779 sys_catalog.cc:458] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:56.271521 28774 sys_catalog.cc:455] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de96dfc1631b4c52b95f2c5f2732064c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de96dfc1631b4c52b95f2c5f2732064c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34921 } } }
I20260806 15:54:56.271701 28774 sys_catalog.cc:458] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:56.271778 28787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:56.274545 28787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:56.280040 28787 catalog_manager.cc:1347] Generated new cluster ID: ea57915e540844ef896df71b20a252f2
I20260806 15:54:56.280107 28787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:56.289417 28787 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:56.290293 28787 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:56.294440 28787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de96dfc1631b4c52b95f2c5f2732064c: Generated new TSK 0
I20260806 15:54:56.295070 28787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:56.301446 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34921
--builtin_ntp_servers=127.13.223.20:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 15:54:56.416013 28802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:56.416301 28802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:56.416379 28802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:56.420574 28802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:56.420650 28802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:56.420732 28802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:56.425431 28802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:56.426596 28802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:56.428014 28802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:56.438017 28818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.437994 28815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.438051 28813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:56.438326 28802 server_base.cc:1034] running on GCE node
I20260806 15:54:56.438839 28802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:56.439455 28802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:56.440598 28802 hybrid_clock.cc:648] HybridClock initialized: now 1786031696440561 us; error 47 us; skew 500 ppm
I20260806 15:54:56.442454 28802 webserver.cc:460] Webserver started at http://127.13.223.1:45935/ using document root <none> and password file <none>
I20260806 15:54:56.442997 28802 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:56.443054 28802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:56.443210 28802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:56.445309 28802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "98c3fa98401942e38bb4eaa3f181fbcc"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "f9136870d25e60092cab1afe858b339a"
server_key_iv: "7dd899c55989064e20473fdf9bde0b41"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.445802 28802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "98c3fa98401942e38bb4eaa3f181fbcc"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "f9136870d25e60092cab1afe858b339a"
server_key_iv: "7dd899c55989064e20473fdf9bde0b41"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.449433 28802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:54:56.452085 28826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.453524 28802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:54:56.453644 28802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "98c3fa98401942e38bb4eaa3f181fbcc"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "f9136870d25e60092cab1afe858b339a"
server_key_iv: "7dd899c55989064e20473fdf9bde0b41"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.453742 28802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:56.466346 28802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:56.466969 28802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:56.467129 28802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:56.467640 28802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:56.468623 28802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:56.468673 28802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.468715 28802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:56.468737 28802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.478089 28802 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:40177
I20260806 15:54:56.478120 29004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:40177 every 8 connection(s)
I20260806 15:54:56.479000 28802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:56.488584 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.480814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:42195 (local address 127.13.223.62:34921)
0806 15:54:56.480948 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:56.480951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:56.482088 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 15:54:56.483611 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:56.483615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:56.483678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:56.483790 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:56.484832 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.485375 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.486540 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.486774 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.486890 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.488021 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:56.488025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:56.488027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:56.488047 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:56.488066 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:56.488074 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:56.488151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:56.488334 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 15:54:56.488404 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:54:56.488879 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 28802
I20260806 15:54:56.488997 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:56.489284 14204 external_mini_cluster.cc:1428] Setting key d339425af8744a23068130d4afa119b0
I20260806 15:54:56.489544 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.481258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34921 (local address 127.13.223.1:42195)
0806 15:54:56.481937 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:56.481979 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:56.483330 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:56.484001 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:56.484008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:56.484111 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:56.484680 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:56.484697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.485520 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.485524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:56.486408 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:56.486417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.486933 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.486937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:56.487116 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.487758 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:56.487759 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:56.487797 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:56.487867 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:56.488165 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:56.488177 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:56.488190 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 15:54:56.488333 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 15:54:56.488626 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260806 15:54:56.490896 29005 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34921
I20260806 15:54:56.491142 29005 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:56.491444 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34921
--builtin_ntp_servers=127.13.223.20:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 15:54:56.491775 29005 heartbeater.cc:511] Master 127.13.223.62:34921 requested a full tablet report, sending...
I20260806 15:54:56.493355 28704 ts_manager.cc:194] Registered new tserver with Master: 98c3fa98401942e38bb4eaa3f181fbcc (127.13.223.1:40177)
I20260806 15:54:56.494664 28704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.1:42195
W20260806 15:54:56.606340 29011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:56.606673 29011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:56.606753 29011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:56.610956 29011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:56.611058 29011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:56.611159 29011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:56.615919 29011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:56.617081 29011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:56.618696 29011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:56.628288 29026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:56.628552 29011 server_base.cc:1034] running on GCE node
W20260806 15:54:56.628293 29024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.628283 29028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:56.629089 29011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:56.629722 29011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:56.630911 29011 hybrid_clock.cc:648] HybridClock initialized: now 1786031696630875 us; error 47 us; skew 500 ppm
I20260806 15:54:56.632896 29011 webserver.cc:460] Webserver started at http://127.13.223.2:46807/ using document root <none> and password file <none>
I20260806 15:54:56.633496 29011 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:56.633616 29011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:56.633877 29011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:56.635984 29011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "ff255a32bc694cc782c049192372d640"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "ebe5a8725c1a76b5e9a361004117936d"
server_key_iv: "a71ed4d814cdf682286a2e76dad02ef9"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.636672 29011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "ff255a32bc694cc782c049192372d640"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "ebe5a8725c1a76b5e9a361004117936d"
server_key_iv: "a71ed4d814cdf682286a2e76dad02ef9"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.640434 29011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:56.643157 29038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.644320 29011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 15:54:56.644461 29011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "ff255a32bc694cc782c049192372d640"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "ebe5a8725c1a76b5e9a361004117936d"
server_key_iv: "a71ed4d814cdf682286a2e76dad02ef9"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.644667 29011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:56.660426 29011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:56.661131 29011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:56.661331 29011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:56.661921 29011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:56.662887 29011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:56.662969 29011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.663065 29011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:56.663120 29011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.672151 29011 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:36971
I20260806 15:54:56.672180 29205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:36971 every 8 connection(s)
I20260806 15:54:56.673137 29011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:56.679123 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 29011
I20260806 15:54:56.679224 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:56.679565 14204 external_mini_cluster.cc:1428] Setting key c1cf825876305c9fc3894b2a6b3db947
I20260806 15:54:56.681680 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34921
--builtin_ntp_servers=127.13.223.20:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 15:54:56.683300 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.674981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:46953 (local address 127.13.223.62:34921)
0806 15:54:56.675143 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:56.675147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:56.676165 (+ 1018us) server_negotiation.cc:407] Connection header received
0806 15:54:56.677601 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:56.677603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:56.677662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:56.677734 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:56.678831 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.679388 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.680921 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.681250 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.681384 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.682694 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:56.682698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:56.682701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:56.682733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:56.682777 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:56.682787 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:56.682870 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:56.683075 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 15:54:56.683145 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 15:54:56.684525 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.675349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34921 (local address 127.13.223.2:46953)
0806 15:54:56.676024 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:56.676065 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:56.677389 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:56.677893 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:56.677900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:56.677976 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:56.678694 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:56.678710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.679530 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.679534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:56.680781 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:56.680792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.681422 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.681427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:56.681609 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.682380 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:56.682381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:56.682422 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:56.682485 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:56.682885 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:56.682902 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:56.682909 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 15:54:56.683074 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 15:54:56.683348 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260806 15:54:56.685894 29206 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34921
I20260806 15:54:56.686143 29206 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:56.686694 29206 heartbeater.cc:511] Master 127.13.223.62:34921 requested a full tablet report, sending...
I20260806 15:54:56.687813 28704 ts_manager.cc:194] Registered new tserver with Master: ff255a32bc694cc782c049192372d640 (127.13.223.2:36971)
I20260806 15:54:56.688369 28704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.2:46953
W20260806 15:54:56.799690 29211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:56.799981 29211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:56.800057 29211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:56.804154 29211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:56.804237 29211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:56.804318 29211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:56.809172 29211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:56.810263 29211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:56.811648 29211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:56.822862 29227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.822901 29223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:56.823321 29229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:56.823598 29211 server_base.cc:1034] running on GCE node
I20260806 15:54:56.824004 29211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:56.824714 29211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:56.825904 29211 hybrid_clock.cc:648] HybridClock initialized: now 1786031696825896 us; error 31 us; skew 500 ppm
I20260806 15:54:56.827821 29211 webserver.cc:460] Webserver started at http://127.13.223.3:38041/ using document root <none> and password file <none>
I20260806 15:54:56.828406 29211 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:56.828500 29211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:56.828768 29211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:56.830933 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "033ac6aa35424803aec2e25ee05e066c"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "60ed9e7e64d92f07cb8e22ce362c968a"
server_key_iv: "f40eb439e0015d92a70fcfa2730e524a"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.831606 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "033ac6aa35424803aec2e25ee05e066c"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "60ed9e7e64d92f07cb8e22ce362c968a"
server_key_iv: "f40eb439e0015d92a70fcfa2730e524a"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.835644 29211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 15:54:56.838376 29239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.839732 29211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 15:54:56.839902 29211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "033ac6aa35424803aec2e25ee05e066c"
format_stamp: "Formatted at 2026-08-06 15:54:56 on dist-test-slave-rwrg"
server_key: "60ed9e7e64d92f07cb8e22ce362c968a"
server_key_iv: "f40eb439e0015d92a70fcfa2730e524a"
server_key_version: "encryptionkey@0"
I20260806 15:54:56.840027 29211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:56.861132 29211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:56.861868 29211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:56.862063 29211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:56.862649 29211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:56.863583 29211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:56.863663 29211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.863754 29211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:56.863812 29211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:56.873427 29211 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:33821
I20260806 15:54:56.873456 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:33821 every 8 connection(s)
I20260806 15:54:56.874351 29211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:56.879959 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 29211
I20260806 15:54:56.880115 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:56.880414 14204 external_mini_cluster.cc:1428] Setting key 4ac7b4544ef3052de1a408e41c06bca0
I20260806 15:54:56.884116 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.876239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:39551 (local address 127.13.223.62:34921)
0806 15:54:56.876380 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:56.876389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:56.877495 (+ 1106us) server_negotiation.cc:407] Connection header received
0806 15:54:56.878932 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:56.878936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:56.878997 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:56.879080 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:56.879963 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.880696 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.882216 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.882480 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.882579 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.883651 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:56.883654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:56.883656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:56.883668 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:56.883685 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:56.883693 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:56.883762 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:56.883954 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 15:54:56.884006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:54:56.885088 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:56.876683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34921 (local address 127.13.223.3:39551)
0806 15:54:56.877336 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:56.877377 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:56.878652 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:56.879257 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:56.879264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:56.879346 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:56.879808 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:56.879820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.880846 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.880869 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:56.882088 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:56.882098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:56.882675 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:56.882681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:56.882850 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:56.883406 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:56.883408 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:56.883446 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:56.883509 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:56.883804 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:56.883813 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:56.883818 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 15:54:56.883934 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 15:54:56.884181 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260806 15:54:56.886184 29408 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34921
I20260806 15:54:56.886420 29408 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:56.886934 29408 heartbeater.cc:511] Master 127.13.223.62:34921 requested a full tablet report, sending...
I20260806 15:54:56.887919 28704 ts_manager.cc:194] Registered new tserver with Master: 033ac6aa35424803aec2e25ee05e066c (127.13.223.3:33821)
I20260806 15:54:56.888496 28704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.3:39551
I20260806 15:54:56.894354 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 15:54:56.897399 28768 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:56.896427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.13.223.62:34921)
0806 15:54:56.896620 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:56.896625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:56.896804 (+ 179us) server_negotiation.cc:407] Connection header received
0806 15:54:56.896879 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:56.896881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:56.896924 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:56.896997 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:56.897274 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33506: BlockingRecv error: recv got EOF from 127.0.0.1:33506 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 15:54:56.897507 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 28802
I20260806 15:54:56.919294 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 29011
I20260806 15:54:56.930600 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 29211
I20260806 15:54:56.946225 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 28657
2026-08-06T15:54:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (906 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T15:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:56Z Disabled control of system clock
I20260806 15:54:56.981094 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43099
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:43099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 15:54:57.096002 29427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:57.096334 29427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:57.096374 29427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:57.100348 29427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:57.100415 29427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:57.100431 29427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:57.100447 29427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:57.100462 29427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:57.105361 29427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:43099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:57.106492 29427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:57.107875 29427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:57.116272 29434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:57.116303 29435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:57.116384 29427 server_base.cc:1034] running on GCE node
W20260806 15:54:57.116302 29437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:57.117141 29427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:57.118178 29427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:57.119354 29427 hybrid_clock.cc:648] HybridClock initialized: now 1786031697119334 us; error 31 us; skew 500 ppm
I20260806 15:54:57.121212 29427 webserver.cc:460] Webserver started at http://127.13.223.62:34839/ using document root <none> and password file <none>
I20260806 15:54:57.121809 29427 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:57.121870 29427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:57.122130 29427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:57.124244 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "afedf26056794aadada706c0d86fa49c"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "dff321dc71ae799c7d8dce9b2ddfdea3"
server_key_iv: "234d35313c648d20eeb7e4b62350a750"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.124823 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "afedf26056794aadada706c0d86fa49c"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "dff321dc71ae799c7d8dce9b2ddfdea3"
server_key_iv: "234d35313c648d20eeb7e4b62350a750"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.128549 29427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:54:57.131016 29444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.132241 29427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:57.132392 29427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "afedf26056794aadada706c0d86fa49c"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "dff321dc71ae799c7d8dce9b2ddfdea3"
server_key_iv: "234d35313c648d20eeb7e4b62350a750"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.132535 29427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:57.137321 29427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:57.137933 29427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:57.138130 29427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:57.145186 29427 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:43099
I20260806 15:54:57.145234 29524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:43099 every 8 connection(s)
I20260806 15:54:57.146180 29427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:57.147188 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 29427
I20260806 15:54:57.147332 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:57.147643 14204 external_mini_cluster.cc:1428] Setting key f5d90bf65b8453b657a7e4b107f5f489
I20260806 15:54:57.150128 29525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:57.153527 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.149082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.13.223.62:43099)
0806 15:54:57.149827 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:57.149843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:57.149877 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:54:57.150590 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:57.150616 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:57.150969 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:57.151314 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:57.152226 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:57.152231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:57.152243 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:57.152254 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:57.152310 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:57.152326 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:57.152485 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:57.152579 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 15:54:57.152838 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":208,"threads_started":1}
I20260806 15:54:57.156580 29525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c: Bootstrap starting.
I20260806 15:54:57.158777 29525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c: Neither blocks nor log segments found. Creating new log.
I20260806 15:54:57.159608 29525 log.cc:824] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:57.161943 29525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c: No bootstrap required, opened a new log
I20260806 15:54:57.165853 29525 raft_consensus.cc:348] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf26056794aadada706c0d86fa49c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43099 } }
I20260806 15:54:57.166091 29525 raft_consensus.cc:374] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:57.166168 29525 raft_consensus.cc:729] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afedf26056794aadada706c0d86fa49c, State: Initialized, Role: FOLLOWER
I20260806 15:54:57.166637 29525 consensus_queue.cc:260] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf26056794aadada706c0d86fa49c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43099 } }
I20260806 15:54:57.166783 29525 raft_consensus.cc:388] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:57.166848 29525 raft_consensus.cc:482] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:57.166947 29525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:57.167870 29525 raft_consensus.cc:504] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf26056794aadada706c0d86fa49c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43099 } }
I20260806 15:54:57.168215 29525 leader_election.cc:302] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afedf26056794aadada706c0d86fa49c; no voters:
I20260806 15:54:57.168602 29525 leader_election.cc:288] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:57.168689 29531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:57.168987 29531 raft_consensus.cc:686] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [term 1 LEADER]: Becoming Leader. State: Replica: afedf26056794aadada706c0d86fa49c, State: Running, Role: LEADER
I20260806 15:54:57.169305 29531 consensus_queue.cc:237] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf26056794aadada706c0d86fa49c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43099 } }
I20260806 15:54:57.169942 29525 sys_catalog.cc:564] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:57.170904 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afedf26056794aadada706c0d86fa49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf26056794aadada706c0d86fa49c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43099 } } }
I20260806 15:54:57.171036 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:57.171432 29545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:57.171483 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [sys.catalog]: SysCatalogTable state changed. Reason: New leader afedf26056794aadada706c0d86fa49c. Latest consensus state: current_term: 1 leader_uuid: "afedf26056794aadada706c0d86fa49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afedf26056794aadada706c0d86fa49c" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43099 } } }
I20260806 15:54:57.171583 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:57.174631 29545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:57.179908 29545 catalog_manager.cc:1347] Generated new cluster ID: 965de7c9d6954034a8d3fe236c27426a
I20260806 15:54:57.179980 29545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:57.192855 29545 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:57.193193 29545 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:57.203871 29545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afedf26056794aadada706c0d86fa49c: Generated new TSK 0
I20260806 15:54:57.204697 29545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:54:57.207600 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43099
--builtin_ntp_servers=127.13.223.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 15:54:57.323668 29564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:57.323982 29564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:57.324054 29564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:57.328059 29564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:57.328111 29564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:57.328222 29564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:57.333222 29564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:57.334420 29564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:57.335940 29564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:57.345829 29573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:57.345836 29572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:57.345826 29579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:57.346055 29564 server_base.cc:1034] running on GCE node
I20260806 15:54:57.346555 29564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:57.347203 29564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:57.348368 29564 hybrid_clock.cc:648] HybridClock initialized: now 1786031697348331 us; error 50 us; skew 500 ppm
I20260806 15:54:57.350258 29564 webserver.cc:460] Webserver started at http://127.13.223.1:46247/ using document root <none> and password file <none>
I20260806 15:54:57.350852 29564 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:57.350916 29564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:57.351176 29564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:57.353338 29564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "a9c858d7b7da488a83ba35eb15a26b83"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "b8e1c3f685055ce2e0799e37c91e9b33"
server_key_iv: "da654ce4bb858a01f89d4c58bacaf00f"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.353943 29564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "a9c858d7b7da488a83ba35eb15a26b83"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "b8e1c3f685055ce2e0799e37c91e9b33"
server_key_iv: "da654ce4bb858a01f89d4c58bacaf00f"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.357659 29564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 15:54:57.360205 29586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.361404 29564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:57.361550 29564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "a9c858d7b7da488a83ba35eb15a26b83"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "b8e1c3f685055ce2e0799e37c91e9b33"
server_key_iv: "da654ce4bb858a01f89d4c58bacaf00f"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.361668 29564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:57.366211 29564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:57.366822 29564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:57.367007 29564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:57.367538 29564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:57.368430 29564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:57.368502 29564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.368611 29564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:57.368666 29564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.377698 29564 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:43381
I20260806 15:54:57.377774 29752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:43381 every 8 connection(s)
I20260806 15:54:57.378647 29564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:54:57.384855 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.380546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:33831 (local address 127.13.223.62:43099)
0806 15:54:57.380677 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:57.380680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:57.381774 (+ 1094us) server_negotiation.cc:407] Connection header received
0806 15:54:57.383030 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:57.383046 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:57.383088 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:57.383176 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:57.384330 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:57.384334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:57.384336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:57.384350 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:57.384371 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:57.384380 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:57.384485 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:57.384677 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 15:54:57.384719 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 15:54:57.385093 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 29564
I20260806 15:54:57.385213 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:57.385519 14204 external_mini_cluster.cc:1428] Setting key 92cbe9dcaf2f76c8ca53b41de334b119
I20260806 15:54:57.385932 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.380958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43099 (local address 127.13.223.1:33831)
0806 15:54:57.381607 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:57.381646 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:57.382810 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:57.383334 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:57.383340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:57.383427 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:57.384045 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:57.384047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:57.384088 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:57.384163 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:57.384473 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:57.384478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:57.384482 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 15:54:57.384687 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 15:54:57.384918 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260806 15:54:57.387307 29753 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43099
I20260806 15:54:57.387574 29753 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:57.387773 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43099
--builtin_ntp_servers=127.13.223.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 15:54:57.388092 29753 heartbeater.cc:511] Master 127.13.223.62:43099 requested a full tablet report, sending...
I20260806 15:54:57.389926 29478 ts_manager.cc:194] Registered new tserver with Master: a9c858d7b7da488a83ba35eb15a26b83 (127.13.223.1:43381)
W20260806 15:54:57.503701 29761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:57.504035 29761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:57.504078 29761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:57.508111 29761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:57.508175 29761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:57.508232 29761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:57.512856 29761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:57.513974 29761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:57.515524 29761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:57.530027 29772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:57.530040 29768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:57.530040 29767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:57.530225 29761 server_base.cc:1034] running on GCE node
I20260806 15:54:57.530701 29761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:57.531332 29761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:57.532578 29761 hybrid_clock.cc:648] HybridClock initialized: now 1786031697532519 us; error 53 us; skew 500 ppm
I20260806 15:54:57.534461 29761 webserver.cc:460] Webserver started at http://127.13.223.2:36519/ using document root <none> and password file <none>
I20260806 15:54:57.535120 29761 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:57.535226 29761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:57.535481 29761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:57.537830 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "8d540515f5604c04bfff10f910e074ec"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "5329875433669d7377ef9785297264a9"
server_key_iv: "a1584b1c841f9cfce09e1d11cc0e9880"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.538491 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "8d540515f5604c04bfff10f910e074ec"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "5329875433669d7377ef9785297264a9"
server_key_iv: "a1584b1c841f9cfce09e1d11cc0e9880"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.542505 29761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:57.545360 29781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.546638 29761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:54:57.546784 29761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "8d540515f5604c04bfff10f910e074ec"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "5329875433669d7377ef9785297264a9"
server_key_iv: "a1584b1c841f9cfce09e1d11cc0e9880"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.546912 29761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:57.551685 29761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:57.552312 29761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:57.552547 29761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:57.553146 29761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:57.554116 29761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:57.554195 29761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.554286 29761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:57.554338 29761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.563066 29761 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:33309
I20260806 15:54:57.563084 29961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:33309 every 8 connection(s)
I20260806 15:54:57.563968 29761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:54:57.565796 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 29761
I20260806 15:54:57.565948 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:54:57.566259 14204 external_mini_cluster.cc:1428] Setting key 7903ad7e194cb7595dc5bdaf03584e83
I20260806 15:54:57.568604 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43099
--builtin_ntp_servers=127.13.223.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 15:54:57.571226 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.566011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:32971 (local address 127.13.223.62:43099)
0806 15:54:57.566184 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:57.566189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:57.567445 (+ 1256us) server_negotiation.cc:407] Connection header received
0806 15:54:57.569283 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:57.569286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:57.569319 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:57.569383 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:57.570637 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:57.570643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:57.570647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:57.570664 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:57.570690 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:57.570704 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:57.570771 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:57.571032 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 15:54:57.571090 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 15:54:57.572247 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.566503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43099 (local address 127.13.223.2:32971)
0806 15:54:57.567293 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:57.567334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:57.569027 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:57.569571 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:57.569581 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:57.569699 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:57.570273 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:57.570278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:57.570334 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:57.570427 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:57.570866 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:57.570878 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:57.570886 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 15:54:57.571119 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 15:54:57.571387 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":149,"threads_started":1}
I20260806 15:54:57.573523 29962 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43099
I20260806 15:54:57.573773 29962 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:57.574234 29962 heartbeater.cc:511] Master 127.13.223.62:43099 requested a full tablet report, sending...
I20260806 15:54:57.575413 29478 ts_manager.cc:194] Registered new tserver with Master: 8d540515f5604c04bfff10f910e074ec (127.13.223.2:33309)
W20260806 15:54:57.685288 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:57.685546 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:57.685585 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:57.689658 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:57.689723 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:57.689781 29966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:54:57.694552 29966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:57.695605 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:57.697153 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:57.707512 29978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:57.707552 29982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:57.707863 29966 server_base.cc:1034] running on GCE node
W20260806 15:54:57.707978 29977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:57.708530 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:57.709211 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:57.710412 29966 hybrid_clock.cc:648] HybridClock initialized: now 1786031697710337 us; error 95 us; skew 500 ppm
I20260806 15:54:57.712846 29966 webserver.cc:460] Webserver started at http://127.13.223.3:33715/ using document root <none> and password file <none>
I20260806 15:54:57.713594 29966 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:57.713689 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:57.713938 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:57.716343 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "63664175e22c49a8866b01642d4182cd"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "3d8a2b63b3157b5bc8f1ab4fd4d4b61b"
server_key_iv: "b53fb3bb0c0adac7365b14bb56c59e7f"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.716982 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "63664175e22c49a8866b01642d4182cd"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "3d8a2b63b3157b5bc8f1ab4fd4d4b61b"
server_key_iv: "b53fb3bb0c0adac7365b14bb56c59e7f"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.720783 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:57.723415 29992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.724704 29966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:57.724802 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "63664175e22c49a8866b01642d4182cd"
format_stamp: "Formatted at 2026-08-06 15:54:57 on dist-test-slave-rwrg"
server_key: "3d8a2b63b3157b5bc8f1ab4fd4d4b61b"
server_key_iv: "b53fb3bb0c0adac7365b14bb56c59e7f"
server_key_version: "encryptionkey@0"
I20260806 15:54:57.724931 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:57.729630 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:57.730242 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:57.730430 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:57.730970 29966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:57.731870 29966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:57.731917 29966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.731980 29966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:57.732014 29966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:57.741089 29966 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40349
I20260806 15:54:57.741195 30180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40349 every 8 connection(s)
I20260806 15:54:57.742017 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:54:57.750314 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 29966
I20260806 15:54:57.750418 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:54:57.750814 14204 external_mini_cluster.cc:1428] Setting key 17a00149993f5171e2db8165fefe9c31
I20260806 15:54:57.751866 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.743940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:59243 (local address 127.13.223.62:43099)
0806 15:54:57.744090 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:57.744093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:57.745231 (+ 1138us) server_negotiation.cc:407] Connection header received
0806 15:54:57.746682 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:57.746685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:57.746730 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:57.746835 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:57.751212 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:57.751217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:57.751221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:54:57.751240 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:57.751272 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:54:57.751282 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:57.751383 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:57.751668 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 15:54:57.751724 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:54:57.753227 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:57.744339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43099 (local address 127.13.223.3:59243)
0806 15:54:57.745073 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:57.745115 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:57.746417 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:57.747004 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:57.747011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:57.747102 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:57.750808 (+ 3706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:54:57.750810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:57.750866 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:54:57.751012 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:57.751423 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:57.751439 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:57.751450 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 15:54:57.751762 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 15:54:57.752111 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":172,"threads_started":1}
I20260806 15:54:57.754812 30181 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43099
I20260806 15:54:57.755115 30181 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:57.755738 30181 heartbeater.cc:511] Master 127.13.223.62:43099 requested a full tablet report, sending...
I20260806 15:54:57.756963 29478 ts_manager.cc:194] Registered new tserver with Master: 63664175e22c49a8866b01642d4182cd (127.13.223.3:40349)
I20260806 15:54:57.765470 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 15:54:57.768759 29528 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:54:57.767721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.13.223.62:43099)
0806 15:54:57.767894 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:57.767899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:57.768097 (+ 198us) server_negotiation.cc:407] Connection header received
0806 15:54:57.768195 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:57.768197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:57.768234 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:57.768302 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:57.768619 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54128: BlockingRecv error: recv got EOF from 127.0.0.1:54128 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 15:54:57.768970 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 29564
I20260806 15:54:57.779650 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 29761
I20260806 15:54:57.799845 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 29966
I20260806 15:54:57.814697 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 29427
2026-08-06T15:54:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (872 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:54:57 dist-test-slave-rwrg krb5kdc[30196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:54:57 dist-test-slave-rwrg krb5kdc[30196](info): set up 2 sockets
Aug 06 15:54:57 dist-test-slave-rwrg krb5kdc[30196](info): commencing operation
krb5kdc: starting...
W20260806 15:54:59.165848 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:54:59 dist-test-slave-rwrg krb5kdc[30196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:54:59.295799 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43629
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:59.409989 30219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:59.410270 30219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:59.410346 30219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:59.414505 30219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:54:59.414577 30219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:59.414597 30219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:59.414613 30219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:54:59.414629 30219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:54:59.420102 30219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:59.421324 30219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:59.422353 30219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:54:59.431279 30219 server_base.cc:1034] running on GCE node
W20260806 15:54:59.431242 30229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:59.431226 30225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:59.431242 30227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:59.432166 30219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:59.433336 30219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:59.434499 30219 hybrid_clock.cc:648] HybridClock initialized: now 1786031699434474 us; error 39 us; skew 500 ppm
Aug 06 15:54:59 dist-test-slave-rwrg krb5kdc[30196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:59.437387 30219 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:54:59.438269 30219 webserver.cc:460] Webserver started at http://127.13.223.62:46235/ using document root <none> and password file <none>
I20260806 15:54:59.439049 30219 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:59.439152 30219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:59.439419 30219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:59.441519 30219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "28b9dc97d5af44fd904707a077a27de4"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "0d4f3f23174b5e4d392c733e9be9549c"
server_key_iv: "8c566823f7b6257bb537cd2788d7f767"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.442111 30219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "28b9dc97d5af44fd904707a077a27de4"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "0d4f3f23174b5e4d392c733e9be9549c"
server_key_iv: "8c566823f7b6257bb537cd2788d7f767"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.445850 30219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:54:59.448285 30247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:59.449621 30219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:54:59.449754 30219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "28b9dc97d5af44fd904707a077a27de4"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "0d4f3f23174b5e4d392c733e9be9549c"
server_key_iv: "8c566823f7b6257bb537cd2788d7f767"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.449864 30219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:59.477478 30219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:59.481022 30219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:59.481240 30219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:59.488863 30219 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:43629
I20260806 15:54:59.489166 30328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:43629 every 8 connection(s)
I20260806 15:54:59.489897 30219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:54:59.492803 30329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:54:59.492901 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 30219
I20260806 15:54:59.493013 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:54:59.493314 14204 external_mini_cluster.cc:1428] Setting key 276515093d61746713065914b1c37eb6
I20260806 15:54:59.500308 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4: Bootstrap starting.
I20260806 15:54:59.502929 30329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4: Neither blocks nor log segments found. Creating new log.
Aug 06 15:54:59 dist-test-slave-rwrg krb5kdc[30196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:59.503790 30329 log.cc:824] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4: Log is configured to *not* fsync() on all Append() calls
I20260806 15:54:59.506327 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4: No bootstrap required, opened a new log
I20260806 15:54:59.509009 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:59.494765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.13.223.62:43629)
0806 15:54:59.495214 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:59.495233 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:59.495280 (+ 47us) server_negotiation.cc:407] Connection header received
0806 15:54:59.496383 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:59.496408 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:59.496879 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:59.497258 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:59.497937 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:59.499511 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:59.500394 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:59.501129 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:59.501283 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:59.503811 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:59.503818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:59.503837 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:59.503871 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:59.506073 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:59.506599 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:59.506605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:59.506609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:59.506741 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:59.507051 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:59.507053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:59.507054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:59.507430 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:59.507578 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:59.507889 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 15:54:59.508223 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260806 15:54:59.510820 30329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9dc97d5af44fd904707a077a27de4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43629 } }
I20260806 15:54:59.511065 30329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:54:59.511153 30329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b9dc97d5af44fd904707a077a27de4, State: Initialized, Role: FOLLOWER
I20260806 15:54:59.511610 30329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9dc97d5af44fd904707a077a27de4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43629 } }
I20260806 15:54:59.511731 30329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:54:59.511790 30329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:54:59.511873 30329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:54:59.512876 30329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9dc97d5af44fd904707a077a27de4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43629 } }
I20260806 15:54:59.513175 30329 leader_election.cc:302] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b9dc97d5af44fd904707a077a27de4; no voters:
I20260806 15:54:59.513439 30329 leader_election.cc:288] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:54:59.513602 30336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:54:59.513827 30336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [term 1 LEADER]: Becoming Leader. State: Replica: 28b9dc97d5af44fd904707a077a27de4, State: Running, Role: LEADER
I20260806 15:54:59.514189 30336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9dc97d5af44fd904707a077a27de4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43629 } }
I20260806 15:54:59.514827 30329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:54:59.515913 30338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b9dc97d5af44fd904707a077a27de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9dc97d5af44fd904707a077a27de4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43629 } } }
I20260806 15:54:59.516043 30338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:59.516444 30348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:54:59.516405 30339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b9dc97d5af44fd904707a077a27de4. Latest consensus state: current_term: 1 leader_uuid: "28b9dc97d5af44fd904707a077a27de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b9dc97d5af44fd904707a077a27de4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43629 } } }
I20260806 15:54:59.517230 30339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4 [sys.catalog]: This master's current role is: LEADER
I20260806 15:54:59.519471 30348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:54:59.525447 30348 catalog_manager.cc:1347] Generated new cluster ID: 0cdb725331f24e3483b9a239838d53a6
I20260806 15:54:59.525524 30348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:54:59.534852 30348 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:54:59.535825 30348 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:54:59.541481 30348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4: Generated new TSK 0
I20260806 15:54:59.542303 30348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:54:59.580042 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43629
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:59.693917 30366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:59.694216 30366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:59.694260 30366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:59.698450 30366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:59.698530 30366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:59.698621 30366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:54:59.704340 30366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:59.705911 30366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:59.707393 30366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:59.718827 30377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:59.718847 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:59.718863 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:59.719154 30366 server_base.cc:1034] running on GCE node
I20260806 15:54:59.719651 30366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:59.720326 30366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:59.721514 30366 hybrid_clock.cc:648] HybridClock initialized: now 1786031699721484 us; error 48 us; skew 500 ppm
Aug 06 15:54:59 dist-test-slave-rwrg krb5kdc[30196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:59.724241 30366 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:54:59.725198 30366 webserver.cc:460] Webserver started at http://127.13.223.1:38263/ using document root <none> and password file <none>
I20260806 15:54:59.725781 30366 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:59.725873 30366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:59.726116 30366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:59.728137 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "bc2339b73a7b771322efa75893c86a00"
server_key_iv: "ea59eadadc64389233948c108bad8dec"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.728758 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "bc2339b73a7b771322efa75893c86a00"
server_key_iv: "ea59eadadc64389233948c108bad8dec"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.732395 30366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:54:59.734887 30390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:59.736109 30366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:54:59.736249 30366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "bc2339b73a7b771322efa75893c86a00"
server_key_iv: "ea59eadadc64389233948c108bad8dec"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.736361 30366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:59.750946 30366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:59.754058 30366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:59.754335 30366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:59.754979 30366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:59.755815 30366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:59.755887 30366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:59.755959 30366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:59.756002 30366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:59.766295 30366 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:37733
I20260806 15:54:59.766350 30559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:37733 every 8 connection(s)
I20260806 15:54:59.767233 30366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
Aug 06 15:54:59 dist-test-slave-rwrg krb5kdc[30196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:54:59.777206 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 30366
I20260806 15:54:59.777361 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:54:59.777748 14204 external_mini_cluster.cc:1428] Setting key 9609139d10515d3908c58d72b9e2402a
I20260806 15:54:59.781662 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:59.769170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:57945 (local address 127.13.223.62:43629)
0806 15:54:59.769348 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 15:54:59.769352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:54:59.770141 (+ 789us) server_negotiation.cc:407] Connection header received
0806 15:54:59.771477 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:54:59.771480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:54:59.771528 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:54:59.771606 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:54:59.772858 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:59.773394 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:59.774515 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:54:59.774756 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:54:59.774835 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:59.777349 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:54:59.777353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:54:59.777356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:54:59.777387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:54:59.779360 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:59.779917 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:59.779924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:59.779929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:59.780036 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:54:59.780371 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:54:59.780376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:54:59.780381 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:54:59.780668 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:54:59.780772 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:54:59.781267 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 15:54:59.781414 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:54:59.782689 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:54:59.769529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43629 (local address 127.13.223.1:57945)
0806 15:54:59.769987 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 15:54:59.770030 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:54:59.771278 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:54:59.771793 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:54:59.771799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:54:59.772160 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:54:59.772720 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:59.772736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:59.773511 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:59.773514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:59.774405 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:54:59.774414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:54:59.774886 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:54:59.774889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:54:59.775025 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:54:59.775630 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:54:59.775646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:54:59.777158 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:54:59.779503 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:59.779509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:59.779522 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:59.779803 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:59.780140 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:54:59.780142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:54:59.780144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:54:59.780276 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:54:59.780784 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:54:59.780790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:54:59.781025 (+ 235us) client_negotiation.cc:769] Sending connection context
0806 15:54:59.781189 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 15:54:59.781482 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260806 15:54:59.784058 30565 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43629
I20260806 15:54:59.784318 30565 heartbeater.cc:464] Registering TS with master...
I20260806 15:54:59.784912 30565 heartbeater.cc:511] Master 127.13.223.62:43629 requested a full tablet report, sending...
I20260806 15:54:59.786477 30275 ts_manager.cc:194] Registered new tserver with Master: 7e9b42b3d8b041b6b9a3319a83a1aa36 (127.13.223.1:37733)
I20260806 15:54:59.787644 30275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:57945
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:54:59.808126 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43629
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:54:59.921916 30572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:54:59.922170 30572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:54:59.922214 30572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:54:59.926249 30572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:54:59.926308 30572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:54:59.926364 30572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:54:59.931433 30572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:54:59.932379 30572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:54:59.933439 30572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:54:59.943342 30592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:59.943342 30585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:54:59.943487 30588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:54:59.943859 30572 server_base.cc:1034] running on GCE node
I20260806 15:54:59.944271 30572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:54:59.944952 30572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:54:59.946164 30572 hybrid_clock.cc:648] HybridClock initialized: now 1786031699946140 us; error 37 us; skew 500 ppm
Aug 06 15:54:59 dist-test-slave-rwrg krb5kdc[30196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:54:59.949572 30572 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:54:59.950565 30572 webserver.cc:460] Webserver started at http://127.13.223.2:42429/ using document root <none> and password file <none>
I20260806 15:54:59.951191 30572 fs_manager.cc:359] Metadata directory not provided
I20260806 15:54:59.951259 30572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:54:59.951526 30572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:54:59.953928 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "8ac78d6ebde3454487a74c6216b1f14f"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "3cf000caa2d4482c21c1ab86338237b2"
server_key_iv: "6c91c0e90250988c68ec60e65e49f076"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.954596 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "8ac78d6ebde3454487a74c6216b1f14f"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "3cf000caa2d4482c21c1ab86338237b2"
server_key_iv: "6c91c0e90250988c68ec60e65e49f076"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.958863 30572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:54:59.961395 30601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:59.962577 30572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:54:59.962728 30572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "8ac78d6ebde3454487a74c6216b1f14f"
format_stamp: "Formatted at 2026-08-06 15:54:59 on dist-test-slave-rwrg"
server_key: "3cf000caa2d4482c21c1ab86338237b2"
server_key_iv: "6c91c0e90250988c68ec60e65e49f076"
server_key_version: "encryptionkey@0"
I20260806 15:54:59.962841 30572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:54:59.989019 30572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:54:59.992172 30572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:54:59.992412 30572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:54:59.993104 30572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:54:59.994043 30572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:54:59.994122 30572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:54:59.994205 30572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:54:59.994252 30572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:00.003772 30572 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:45061
I20260806 15:55:00.003796 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:45061 every 8 connection(s)
I20260806 15:55:00.004803 30572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:55:00.005210 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 30572
I20260806 15:55:00.005335 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:55:00.005686 14204 external_mini_cluster.cc:1428] Setting key 16da2ae088fe62060beb81ac19a81d98
Aug 06 15:55:00 dist-test-slave-rwrg krb5kdc[30196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031699, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:55:00.023970 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.006813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:45625 (local address 127.13.223.62:43629)
0806 15:55:00.006943 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.006946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.008108 (+ 1162us) server_negotiation.cc:407] Connection header received
0806 15:55:00.009696 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.009702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.009774 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.009836 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:00.011767 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.012311 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.014242 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.014473 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.014571 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.017865 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:00.017868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:00.017870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:00.017896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:00.019473 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:00.020117 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:00.020122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:00.020126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:00.020254 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:00.020753 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:00.020762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:00.020764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:00.021025 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:00.021156 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.023660 (+ 2504us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.023815 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 15:55:00.025349 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.007385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43629 (local address 127.13.223.2:45625)
0806 15:55:00.007921 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.007975 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.009411 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.010292 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.010303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.010864 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:55:00.011588 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.011612 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.012459 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.012466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.014086 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.014112 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.014650 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.014658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.014944 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.015681 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:55:00.015704 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:55:00.017678 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:55:00.019630 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:00.019636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:00.019649 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:00.019956 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:00.020371 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:00.020376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:00.020378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:00.020627 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:00.022791 (+ 2164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:55:00.022803 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:55:00.023203 (+ 400us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.023482 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.023980 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":150,"threads_started":1}
I20260806 15:55:00.026811 30776 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43629
I20260806 15:55:00.027096 30776 heartbeater.cc:464] Registering TS with master...
I20260806 15:55:00.027623 30776 heartbeater.cc:511] Master 127.13.223.62:43629 requested a full tablet report, sending...
I20260806 15:55:00.028841 30275 ts_manager.cc:194] Registered new tserver with Master: 8ac78d6ebde3454487a74c6216b1f14f (127.13.223.2:45061)
I20260806 15:55:00.029889 30275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:45625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:55:00.040150 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43629
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:55:00.155911 30782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:55:00.156163 30782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:55:00.156206 30782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:55:00.160230 30782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:55:00.160297 30782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:55:00.160355 30782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:55:00.165562 30782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:55:00.166716 30782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:55:00.167789 30782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:55:00.177472 30791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:00.177460 30787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:00.177559 30782 server_base.cc:1034] running on GCE node
W20260806 15:55:00.177650 30788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:00.178174 30782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:55:00.178835 30782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:55:00.180052 30782 hybrid_clock.cc:648] HybridClock initialized: now 1786031700180022 us; error 41 us; skew 500 ppm
Aug 06 15:55:00 dist-test-slave-rwrg krb5kdc[30196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031700, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:55:00.183079 30782 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:55:00.183961 30782 webserver.cc:460] Webserver started at http://127.13.223.3:45855/ using document root <none> and password file <none>
I20260806 15:55:00.184593 30782 fs_manager.cc:359] Metadata directory not provided
I20260806 15:55:00.184705 30782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:55:00.184937 30782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:55:00.187000 30782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "68888e824317477c8fa28da56742a7ca"
format_stamp: "Formatted at 2026-08-06 15:55:00 on dist-test-slave-rwrg"
server_key: "9c5315891ea25954bd82d5b5fcde5c64"
server_key_iv: "816e74c44f01d14abc00aab2464341d1"
server_key_version: "encryptionkey@0"
I20260806 15:55:00.187572 30782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "68888e824317477c8fa28da56742a7ca"
format_stamp: "Formatted at 2026-08-06 15:55:00 on dist-test-slave-rwrg"
server_key: "9c5315891ea25954bd82d5b5fcde5c64"
server_key_iv: "816e74c44f01d14abc00aab2464341d1"
server_key_version: "encryptionkey@0"
I20260806 15:55:00.191709 30782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:55:00.194239 30803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:00.195482 30782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 15:55:00.195649 30782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "68888e824317477c8fa28da56742a7ca"
format_stamp: "Formatted at 2026-08-06 15:55:00 on dist-test-slave-rwrg"
server_key: "9c5315891ea25954bd82d5b5fcde5c64"
server_key_iv: "816e74c44f01d14abc00aab2464341d1"
server_key_version: "encryptionkey@0"
I20260806 15:55:00.195765 30782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:55:00.224099 30782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:55:00.227468 30782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:55:00.227707 30782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:55:00.228406 30782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:55:00.229372 30782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:55:00.229452 30782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:00.229554 30782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:55:00.229617 30782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:00.239928 30782 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:45297
I20260806 15:55:00.239969 30984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:45297 every 8 connection(s)
I20260806 15:55:00.241047 30782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:55:00.248243 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 30782
I20260806 15:55:00.248409 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:55:00.248857 14204 external_mini_cluster.cc:1428] Setting key b6793fa33488737e97a8ff9fd6f4764e
Aug 06 15:55:00 dist-test-slave-rwrg krb5kdc[30196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031700, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:55:00.256096 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.243045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:48939 (local address 127.13.223.62:43629)
0806 15:55:00.243251 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.243256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.244163 (+ 907us) server_negotiation.cc:407] Connection header received
0806 15:55:00.245707 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.245710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.245759 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.245834 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:00.247036 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.247571 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.248937 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.249305 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.249442 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.252323 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:00.252326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:00.252328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:00.252353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:00.253983 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:00.254596 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:00.254599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:00.254600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:00.254688 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:00.255034 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:00.255037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:00.255038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:00.255227 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:00.255309 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.255862 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.255975 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 15:55:00.257157 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.243435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43629 (local address 127.13.223.3:48939)
0806 15:55:00.243943 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.244026 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.245466 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.246002 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.246009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.246381 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:55:00.246903 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.246917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.247698 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.247701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.248779 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.248790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.249501 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.249518 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.249704 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.250429 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:55:00.250451 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:55:00.252156 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:55:00.254098 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:00.254103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:00.254117 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:00.254479 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:00.254787 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:00.254789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:00.254791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:00.254939 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:00.255325 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:55:00.255330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:55:00.255593 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.255800 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.256136 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260806 15:55:00.258391 30986 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43629
I20260806 15:55:00.258661 30986 heartbeater.cc:464] Registering TS with master...
I20260806 15:55:00.259183 30986 heartbeater.cc:511] Master 127.13.223.62:43629 requested a full tablet report, sending...
I20260806 15:55:00.260234 30275 ts_manager.cc:194] Registered new tserver with Master: 68888e824317477c8fa28da56742a7ca (127.13.223.3:45297)
I20260806 15:55:00.260938 30275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:48939
I20260806 15:55:00.263139 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:55:00.272729 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.265222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.13.223.62:43629)
0806 15:55:00.265385 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.265390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.265640 (+ 250us) server_negotiation.cc:407] Connection header received
0806 15:55:00.265704 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.265706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.265745 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.265833 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:00.266567 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.267164 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.267954 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.268181 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.268291 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.269643 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:00.269645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:00.269647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:00.269669 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:00.271154 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:00.271576 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:00.271591 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:00.271592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:00.271665 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:00.271894 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:00.271897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:00.271898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:00.272095 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:00.272190 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.272385 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.272474 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:55:00.275543 30275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:55:00.278784 30275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:55:00.293663 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.288005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.13.223.3:45297)
0806 15:55:00.288726 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.288741 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.288764 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:55:00.288830 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.288834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.289013 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.289165 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.289867 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.290609 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.292221 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.292936 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.293064 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.293228 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.293404 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.293499 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":175,"threads_started":1}
I20260806 15:55:00.293660 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.287385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:45297 (local address 127.0.0.1:33746)
0806 15:55:00.288607 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.288622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.288712 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.289366 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.289372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.289415 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.289722 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.289730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.290740 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.290745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.292083 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.292094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.293070 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.293074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.293194 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.293231 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.293371 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.293492 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":37,"spinlock_wait_cycles":15360,"thread_start_us":108,"threads_started":1}
I20260806 15:55:00.294246 31009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.287606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.13.223.2:45061)
0806 15:55:00.288002 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.288015 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.288425 (+ 410us) server_negotiation.cc:407] Connection header received
0806 15:55:00.288802 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.288808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.289013 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.289143 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.289896 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.291002 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.292414 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.293217 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.293309 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.293434 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.293963 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.294064 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":95,"threads_started":1}
I20260806 15:55:00.294440 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.288127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.13.223.1:37733)
0806 15:55:00.288526 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.288533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.288955 (+ 422us) server_negotiation.cc:407] Connection header received
0806 15:55:00.289101 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.289105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.289246 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.289408 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.289776 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.290966 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.292349 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.293186 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.293302 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.293434 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.294198 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.294294 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260806 15:55:00.294667 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.287320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:45061 (local address 127.0.0.1:58698)
0806 15:55:00.288271 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.288325 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.288575 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.289369 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.289374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.289416 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.289722 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.289730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.291290 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.291293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.292306 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.292318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.293501 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.293504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.293687 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.293732 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.294455 (+ 723us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.294537 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":188,"spinlock_wait_cycles":20352,"thread_start_us":91,"threads_started":1}
I20260806 15:55:00.296862 30883 tablet_service.cc:1467] Processing CreateTablet for tablet 250710b586a64904a7f1e14a9a0b0700 (DEFAULT_TABLE table=test-table [id=fbf5dfd5bac948ea86175a9fe0f070d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:55:00.297122 30666 tablet_service.cc:1467] Processing CreateTablet for tablet 250710b586a64904a7f1e14a9a0b0700 (DEFAULT_TABLE table=test-table [id=fbf5dfd5bac948ea86175a9fe0f070d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:55:00.297804 30883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250710b586a64904a7f1e14a9a0b0700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:00.297919 30666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250710b586a64904a7f1e14a9a0b0700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:00.299117 31007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.287663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:37733 (local address 127.0.0.1:51664)
0806 15:55:00.288855 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.288865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.288969 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.289389 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.289392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.289424 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.289663 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.289669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.291092 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.291104 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.292238 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.292246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.293501 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.293504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.293682 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.294063 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.294143 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.294220 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":111,"spinlock_wait_cycles":10496,"thread_start_us":87,"threads_started":1}
I20260806 15:55:00.302228 30464 tablet_service.cc:1467] Processing CreateTablet for tablet 250710b586a64904a7f1e14a9a0b0700 (DEFAULT_TABLE table=test-table [id=fbf5dfd5bac948ea86175a9fe0f070d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:55:00.303349 30464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250710b586a64904a7f1e14a9a0b0700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:00.305308 31014 tablet_bootstrap.cc:492] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Bootstrap starting.
I20260806 15:55:00.305819 31015 tablet_bootstrap.cc:492] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Bootstrap starting.
I20260806 15:55:00.307660 31014 tablet_bootstrap.cc:654] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:00.308393 31015 tablet_bootstrap.cc:654] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:00.309135 31014 log.cc:824] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:00.309145 31015 log.cc:824] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:00.309814 31018 tablet_bootstrap.cc:492] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36: Bootstrap starting.
I20260806 15:55:00.311781 31015 tablet_bootstrap.cc:492] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: No bootstrap required, opened a new log
I20260806 15:55:00.311784 31014 tablet_bootstrap.cc:492] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: No bootstrap required, opened a new log
I20260806 15:55:00.311866 31018 tablet_bootstrap.cc:654] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:00.311977 31014 ts_tablet_manager.cc:1397] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 15:55:00.311977 31015 ts_tablet_manager.cc:1397] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:55:00.312719 31018 log.cc:824] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:00.314364 31018 tablet_bootstrap.cc:492] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36: No bootstrap required, opened a new log
I20260806 15:55:00.314565 31018 ts_tablet_manager.cc:1397] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 15:55:00.316114 31015 raft_consensus.cc:348] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.316119 31014 raft_consensus.cc:348] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.316386 31014 raft_consensus.cc:374] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:00.316386 31015 raft_consensus.cc:374] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:00.316447 31014 raft_consensus.cc:729] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68888e824317477c8fa28da56742a7ca, State: Initialized, Role: FOLLOWER
I20260806 15:55:00.316454 31015 raft_consensus.cc:729] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac78d6ebde3454487a74c6216b1f14f, State: Initialized, Role: FOLLOWER
I20260806 15:55:00.316992 31015 consensus_queue.cc:260] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.316992 31014 consensus_queue.cc:260] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.318032 31015 ts_tablet_manager.cc:1428] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 15:55:00.318032 31014 ts_tablet_manager.cc:1428] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 15:55:00.318434 30776 heartbeater.cc:503] Master 127.13.223.62:43629 was elected leader, sending a full tablet report...
I20260806 15:55:00.318608 30986 heartbeater.cc:503] Master 127.13.223.62:43629 was elected leader, sending a full tablet report...
I20260806 15:55:00.318877 31018 raft_consensus.cc:348] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.319110 31018 raft_consensus.cc:374] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:00.319166 31018 raft_consensus.cc:729] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9b42b3d8b041b6b9a3319a83a1aa36, State: Initialized, Role: FOLLOWER
I20260806 15:55:00.319646 31018 consensus_queue.cc:260] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.320341 30565 heartbeater.cc:503] Master 127.13.223.62:43629 was elected leader, sending a full tablet report...
I20260806 15:55:00.320710 31018 ts_tablet_manager.cc:1428] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260806 15:55:00.355412 31030 raft_consensus.cc:482] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:55:00.355711 31030 raft_consensus.cc:504] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.357579 31027 raft_consensus.cc:482] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:55:00.357769 31027 raft_consensus.cc:504] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.357445 31030 leader_election.cc:288] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac78d6ebde3454487a74c6216b1f14f (127.13.223.2:45061), 68888e824317477c8fa28da56742a7ca (127.13.223.3:45297)
I20260806 15:55:00.358999 31027 leader_election.cc:288] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68888e824317477c8fa28da56742a7ca (127.13.223.3:45297), 7e9b42b3d8b041b6b9a3319a83a1aa36 (127.13.223.1:37733)
I20260806 15:55:00.362948 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.358114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:39155 (local address 127.13.223.3:45297)
0806 15:55:00.358246 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.358251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.358266 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:55:00.358465 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.358469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.358523 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.358641 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.358945 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.359756 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.361637 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.362210 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.362301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.362350 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.362699 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.362792 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:55:00.362955 31009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.357700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:41941 (local address 127.13.223.2:45061)
0806 15:55:00.357827 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.357831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.358206 (+ 375us) server_negotiation.cc:407] Connection header received
0806 15:55:00.358490 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.358493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.358542 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.358591 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.359033 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.359762 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.361460 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.362248 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.362360 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.362405 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.362716 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.362807 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":46}
I20260806 15:55:00.363348 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.357667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:45297 (local address 127.13.223.1:39155)
0806 15:55:00.358158 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.358175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.358320 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.358657 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.358661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.358679 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.358813 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.358820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.360166 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.360172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.361488 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.361501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.362447 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.362450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.362548 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.362569 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.363104 (+ 535us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.363183 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"spinlock_wait_cycles":54912,"thread_start_us":137,"threads_started":1}
I20260806 15:55:00.363365 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.357650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:45061 (local address 127.13.223.1:41941)
0806 15:55:00.358104 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.358120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.358323 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.358752 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.358755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.358771 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.358902 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.358910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.359908 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.359913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.361328 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.361335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.362472 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.362476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.362585 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.362606 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.363148 (+ 542us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.363218 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":32,"spinlock_wait_cycles":13312,"thread_start_us":145,"threads_started":1}
I20260806 15:55:00.364022 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac78d6ebde3454487a74c6216b1f14f" is_pre_election: true
I20260806 15:55:00.364269 30695 raft_consensus.cc:2454] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in term 0.
I20260806 15:55:00.364290 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68888e824317477c8fa28da56742a7ca" is_pre_election: true
I20260806 15:55:00.364643 30912 raft_consensus.cc:2454] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in term 0.
I20260806 15:55:00.364750 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.359146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:45297 (local address 127.13.223.2:34379)
0806 15:55:00.359591 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.359618 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.359700 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.360222 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.360225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.360242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.360345 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.360354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.361525 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.361531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.362773 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.362782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.364295 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.364301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.364423 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.364445 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.364498 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.364613 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":4}
I20260806 15:55:00.364847 30393 leader_election.cc:302] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9b42b3d8b041b6b9a3319a83a1aa36, 8ac78d6ebde3454487a74c6216b1f14f; no voters:
I20260806 15:55:00.365180 31030 raft_consensus.cc:2781] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:55:00.365278 31030 raft_consensus.cc:482] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:55:00.365315 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.359501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:34379 (local address 127.13.223.3:45297)
0806 15:55:00.359949 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.359967 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.359984 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:55:00.360029 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.360034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.360076 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.360448 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.360583 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.361393 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.362909 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.364070 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.365012 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.365069 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.365108 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.365174 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":129,"threads_started":1}
I20260806 15:55:00.365334 31030 raft_consensus.cc:3037] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:00.365670 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.359955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:38835 (local address 127.13.223.1:37733)
0806 15:55:00.360065 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.360069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.360103 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:55:00.360138 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.360141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.360179 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.360245 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:00.360647 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.361321 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.362951 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.363511 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.363638 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.363689 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.364369 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.364438 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:55:00.365757 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "8ac78d6ebde3454487a74c6216b1f14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68888e824317477c8fa28da56742a7ca" is_pre_election: true
I20260806 15:55:00.365867 30912 raft_consensus.cc:2454] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac78d6ebde3454487a74c6216b1f14f in term 0.
I20260806 15:55:00.366035 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.359287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:37733 (local address 127.13.223.2:38835)
0806 15:55:00.359948 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.359962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:00.360052 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:00.360336 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:00.360342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:00.360367 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:00.360479 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.360485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.361460 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.361463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.362828 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:00.362836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.364114 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.364117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:00.364209 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.364232 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:55:00.365810 (+ 1578us) client_negotiation.cc:240] Negotiation successful
0806 15:55:00.365890 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":140,"thread_start_us":120,"threads_started":1}
I20260806 15:55:00.366488 30605 leader_election.cc:302] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68888e824317477c8fa28da56742a7ca, 8ac78d6ebde3454487a74c6216b1f14f; no voters:
I20260806 15:55:00.366627 30488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "8ac78d6ebde3454487a74c6216b1f14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" is_pre_election: true
I20260806 15:55:00.366803 31027 raft_consensus.cc:2781] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:55:00.366796 31030 raft_consensus.cc:504] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.366883 31027 raft_consensus.cc:482] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:55:00.366953 31027 raft_consensus.cc:3037] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:00.367059 30488 raft_consensus.cc:2379] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ac78d6ebde3454487a74c6216b1f14f in current term 1: Already voted for candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in this term.
I20260806 15:55:00.367208 31030 leader_election.cc:288] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac78d6ebde3454487a74c6216b1f14f (127.13.223.2:45061), 68888e824317477c8fa28da56742a7ca (127.13.223.3:45297)
I20260806 15:55:00.367614 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac78d6ebde3454487a74c6216b1f14f"
I20260806 15:55:00.367630 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68888e824317477c8fa28da56742a7ca"
I20260806 15:55:00.367733 30912 raft_consensus.cc:3037] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:00.368302 31027 raft_consensus.cc:504] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.368604 30695 raft_consensus.cc:2379] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in current term 1: Already voted for candidate 8ac78d6ebde3454487a74c6216b1f14f in this term.
I20260806 15:55:00.368700 31027 leader_election.cc:288] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [CANDIDATE]: Term 1 election: Requested vote from peers 68888e824317477c8fa28da56742a7ca (127.13.223.3:45297), 7e9b42b3d8b041b6b9a3319a83a1aa36 (127.13.223.1:37733)
I20260806 15:55:00.369024 30488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "8ac78d6ebde3454487a74c6216b1f14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36"
I20260806 15:55:00.369124 30912 raft_consensus.cc:2454] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in term 1.
I20260806 15:55:00.369184 30488 raft_consensus.cc:2379] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ac78d6ebde3454487a74c6216b1f14f in current term 1: Already voted for candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in this term.
I20260806 15:55:00.369350 30911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250710b586a64904a7f1e14a9a0b0700" candidate_uuid: "8ac78d6ebde3454487a74c6216b1f14f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68888e824317477c8fa28da56742a7ca"
I20260806 15:55:00.369521 30911 raft_consensus.cc:2379] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ac78d6ebde3454487a74c6216b1f14f in current term 1: Already voted for candidate 7e9b42b3d8b041b6b9a3319a83a1aa36 in this term.
I20260806 15:55:00.369443 30394 leader_election.cc:302] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68888e824317477c8fa28da56742a7ca, 7e9b42b3d8b041b6b9a3319a83a1aa36; no voters: 8ac78d6ebde3454487a74c6216b1f14f
I20260806 15:55:00.369746 31030 raft_consensus.cc:2781] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:55:00.369899 30605 leader_election.cc:302] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac78d6ebde3454487a74c6216b1f14f; no voters: 68888e824317477c8fa28da56742a7ca, 7e9b42b3d8b041b6b9a3319a83a1aa36
I20260806 15:55:00.370000 31030 raft_consensus.cc:686] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9b42b3d8b041b6b9a3319a83a1aa36, State: Running, Role: LEADER
I20260806 15:55:00.370137 31027 raft_consensus.cc:2726] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 15:55:00.370354 31030 consensus_queue.cc:237] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } }
I20260806 15:55:00.373824 30273 catalog_manager.cc:5495] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e9b42b3d8b041b6b9a3319a83a1aa36 (127.13.223.1). New cstate: current_term: 1 leader_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9b42b3d8b041b6b9a3319a83a1aa36" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37733 } health_report { overall_health: HEALTHY } } }
I20260806 15:55:00.393607 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.390038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.13.223.1:37733)
0806 15:55:00.390189 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:00.390193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:00.390208 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:55:00.390339 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:00.390341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:00.390395 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:00.390506 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:55:00.390759 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.391257 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.392125 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:00.392344 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:00.392448 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:00.392806 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:55:00.393225 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:55:00.393361 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:00.393408 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 15:55:00.393460 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 15:55:00.399680 30695 raft_consensus.cc:1261] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 1 FOLLOWER]: Refusing update from remote peer 7e9b42b3d8b041b6b9a3319a83a1aa36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:55:00.399680 30911 raft_consensus.cc:1261] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 1 FOLLOWER]: Refusing update from remote peer 7e9b42b3d8b041b6b9a3319a83a1aa36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:55:00.400408 31037 consensus_queue.cc:1035] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68888e824317477c8fa28da56742a7ca" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 45297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:55:00.400723 31030 consensus_queue.cc:1035] T 250710b586a64904a7f1e14a9a0b0700 P 7e9b42b3d8b041b6b9a3319a83a1aa36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac78d6ebde3454487a74c6216b1f14f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:55:00.409637 31042 mvcc.cc:205] Tried to move back new op lower bound from 7315585844829523968 to 7315585844719734784. Current Snapshot: MvccSnapshot[applied={T|T < 7315585844829523968}]
I20260806 15:55:00.410408 31043 mvcc.cc:205] Tried to move back new op lower bound from 7315585844829523968 to 7315585844719734784. Current Snapshot: MvccSnapshot[applied={T|T < 7315585844829523968 or (T in {7315585844829523968})}]
I20260806 15:55:00.417999 30275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:55:00.418201 30275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:55:00.421078 30275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28b9dc97d5af44fd904707a077a27de4: Sending DeleteTablet for 3 replicas of tablet 250710b586a64904a7f1e14a9a0b0700
I20260806 15:55:00.422016 30883 tablet_service.cc:1514] Processing DeleteTablet for tablet 250710b586a64904a7f1e14a9a0b0700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:55:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33746
I20260806 15:55:00.422366 30464 tablet_service.cc:1514] Processing DeleteTablet for tablet 250710b586a64904a7f1e14a9a0b0700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:55:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:51664
I20260806 15:55:00.422573 30666 tablet_service.cc:1514] Processing DeleteTablet for tablet 250710b586a64904a7f1e14a9a0b0700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:55:00 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260806 15:55:00.423120 31061 tablet_replica.cc:330] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: stopping tablet replica
I20260806 15:55:00.423082 31058 tablet_replica.cc:330] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: stopping tablet replica
I20260806 15:55:00.423436 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 30366
I20260806 15:55:00.423508 31061 raft_consensus.cc:2229] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:55:00.428321 31061 raft_consensus.cc:2258] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:55:00.434985 31058 raft_consensus.cc:2229] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:55:00.435474 31058 raft_consensus.cc:2258] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:55:00.437364 31061 ts_tablet_manager.cc:1905] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:55:00.437363 31058 ts_tablet_manager.cc:1905] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:55:00.440718 31061 ts_tablet_manager.cc:1918] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:55:00.440718 31058 ts_tablet_manager.cc:1918] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:55:00.440831 31061 log.cc:1196] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/250710b586a64904a7f1e14a9a0b0700
I20260806 15:55:00.440840 31058 log.cc:1196] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/250710b586a64904a7f1e14a9a0b0700
I20260806 15:55:00.441207 31058 ts_tablet_manager.cc:1939] T 250710b586a64904a7f1e14a9a0b0700 P 68888e824317477c8fa28da56742a7ca: Deleting consensus metadata
I20260806 15:55:00.441208 31061 ts_tablet_manager.cc:1939] T 250710b586a64904a7f1e14a9a0b0700 P 8ac78d6ebde3454487a74c6216b1f14f: Deleting consensus metadata
I20260806 15:55:00.442696 30251 catalog_manager.cc:4841] TS 68888e824317477c8fa28da56742a7ca (127.13.223.3:45297): tablet 250710b586a64904a7f1e14a9a0b0700 (table test-table [id=fbf5dfd5bac948ea86175a9fe0f070d8]) successfully deleted
I20260806 15:55:00.442696 30249 catalog_manager.cc:4841] TS 8ac78d6ebde3454487a74c6216b1f14f (127.13.223.2:45061): tablet 250710b586a64904a7f1e14a9a0b0700 (table test-table [id=fbf5dfd5bac948ea86175a9fe0f070d8]) successfully deleted
W20260806 15:55:00.443120 30248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:37733 (error 108)
I20260806 15:55:00.443969 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 30572
I20260806 15:55:00.444164 31007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:00.443852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:37733 (local address 127.0.0.1:51694)
0806 15:55:00.443978 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:00.444056 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 15:55:00.444320 30248 catalog_manager.cc:4584] TS 7e9b42b3d8b041b6b9a3319a83a1aa36 (127.13.223.1:37733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 250710b586a64904a7f1e14a9a0b0700: Network error: Client connection negotiation failed: client connection to 127.13.223.1:37733: connect: Connection refused (error 111)
I20260806 15:55:00.455464 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 30782
I20260806 15:55:00.473577 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 30219
2026-08-06T15:55:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2656 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:55:00 dist-test-slave-rwrg krb5kdc[31069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:55:00 dist-test-slave-rwrg krb5kdc[31069](info): set up 2 sockets
Aug 06 15:55:00 dist-test-slave-rwrg krb5kdc[31069](info): commencing operation
krb5kdc: starting...
W20260806 15:55:01.764187 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:55:01 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:55:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/rangeradmin_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/rangeradmin_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/rangerlookup_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/rangerlookup_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
I20260806 15:55:01.945107 14204 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBrJzxO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/postgres -l logfile start
2026-08-06 15:55:04.502 UTC [31115] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 15:55:04.502 UTC [31115] LOG: listening on IPv4 address "127.13.223.20", port 51095
2026-08-06 15:55:04.506 UTC [31115] LOG: listening on Unix socket "/tmp/.s.PGSQL.51095"
2026-08-06 15:55:04.528 UTC [31118] LOG: database system was shut down at 2026-08-06 15:55:03 UTC
2026-08-06 15:55:04.532 UTC [31115] LOG: database system is ready to accept connections
I20260806 15:55:05.727226 14204 mini_postgres.cc:96] Postgres bound to 51095
2026-08-06 15:55:05.735 UTC [31134] FATAL: database "slave" does not exist
127.13.223.20:51095 - accepting connections
I20260806 15:55:05.736763 14204 mini_ranger.cc:161] Starting Ranger...
I20260806 15:55:05.753832 14204 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 15:55:06.219139 14204 mini_ranger.cc:87] Created ranger Postgres database
I20260806 15:55:06.219242 14204 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-admin
2026-08-06 15:55:06,613 [I] DB FLAVOR :POSTGRES
2026-08-06 15:55:06,613 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 15:55:06,613 [I] Checking connection..
2026-08-06 15:55:06,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 15:55:06,900 [I] Checking connection passed.
2026-08-06 15:55:06,901 [I] --------- Verifying version history table ---------
2026-08-06 15:55:06,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 15:55:07,167 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 15:55:07,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 15:55:07,447 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 15:55:07,448 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 15:55:07,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 15:55:07.701 UTC [31232] WARNING: there is no transaction in progress
2026-08-06 15:55:07,714 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 15:55:07,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 15:55:07,995 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 15:55:07,995 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 15:55:07,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 15:55:08,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 15:55:08,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rwrg.c.gcp-upstream.internal','N') ;"
2026-08-06 15:55:08,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 15:55:09,080 [I] Table x_portal_user does not exist in database ranger
2026-08-06 15:55:09,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 15:55:09,350 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 15:55:09,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 15:55:09,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 15:55:09,869 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 15:55:09,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 15:55:10.835 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.851 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.866 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.881 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.898 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.915 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.969 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.976 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:10.990 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.000 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.010 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.024 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.034 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.040 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.046 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11.053 UTC [31403] WARNING: there is no transaction in progress
2026-08-06 15:55:11,702 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 15:55:11,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 15:55:11,938 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 15:55:11,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 15:55:12,205 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 15:55:12,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 15:55:12,446 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 15:55:12,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 15:55:12,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 15:55:12,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rwrg.c.gcp-upstream.internal';"
2026-08-06 15:55:13,211 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 15:55:13,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 15:55:13,470 [I] DB_PATCHES have already been applied
I20260806 15:55:13.475859 14204 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBrJzxO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 15:55:13.475925 14204 mini_ranger.cc:193] Using host: 127.13.223.20
I20260806 15:55:13.480383 14204 mini_ranger.cc:227] Ranger admin URL: http://127.13.223.20:51231
Aug 06, 2026 3:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 3:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 3:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 3:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 3:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 15:55:14.910086 14204 mini_ranger.cc:160] Time spent starting Ranger: real 9.173s user 0.005s sys 0.009s
Aug 06, 2026 3:55:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 3:55:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 3:55:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 3:55:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 3:55:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 15:55:38.183724 14204 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/rangerkms_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/rangerkms_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.223.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 15:55:38 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031738, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 15:55:38.276341 14204 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 15:55:38.289325 14204 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 15:55:39.147866 14204 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 15:55:39.147979 14204 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms
I20260806 15:55:39.148097 14204 mini_ranger_kms.cc:227] Using postgres at 127.13.223.20:51095
2026-08-06 15:55:39,797 [I] DB FLAVOR :POSTGRES
2026-08-06 15:55:39,797 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 15:55:39,797 [I] Checking connection
2026-08-06 15:55:39,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 15:55:40,132 [I] connection success
2026-08-06 15:55:40,132 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 15:55:40,132 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 15:55:40,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 15:55:40,394 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 15:55:40,394 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 15:55:40,394 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 15:55:40,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:51095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 15:55:40,702 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 15:55:41.186347 14204 mini_ranger_kms.cc:310] Created kms service
I20260806 15:55:41.368541 14204 mini_ranger_kms.cc:326] Created kudu user
I20260806 15:55:41.417449 14204 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 15:55:41 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031738, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:55:41.849828 14204 mini_ranger_kms.cc:384] Added ranger policy
I20260806 15:55:41.850005 14204 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 15:55:41.850033 14204 mini_ranger_kms.cc:240] Using host: 127.13.223.20
I20260806 15:55:41.854770 14204 mini_ranger_kms.cc:276] Ranger KMS PID: 31688
I20260806 15:55:41.854933 14204 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.223.20:36745
Aug 06, 2026 3:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 3:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 3:55:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 3:55:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 15:55:44.735790 14204 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.459s user 0.000s sys 0.618s
I20260806 15:55:44.735945 14204 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 15:55:44.736101 14204 mini_ranger_kms.cc:401] 127.13.223.20:36745/kms/v1/keys
Aug 06 15:55:44 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786031744, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 15:55:45 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031744, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.20@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
Aug 06 15:55:45 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031744, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.20@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
Aug 06 15:55:48 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:55:48.383497 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40213
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:40213
--ranger_config_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:36745/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:55:48.559368 31771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:55:48.559701 31771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:55:48.559746 31771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:55:48.564327 31771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:55:48.564396 31771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:55:48.564416 31771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:55:48.564433 31771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:55:48.564450 31771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:55:48.570524 31771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.223.20:36745/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:40213
--ranger_config_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:55:48.571722 31771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:55:48.572868 31771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:55:48.584478 31785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:48.584620 31779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:48.584615 31780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:48.584995 31771 server_base.cc:1034] running on GCE node
I20260806 15:55:48.585990 31771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:55:48.587411 31771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:55:48.588584 31771 hybrid_clock.cc:648] HybridClock initialized: now 1786031748588547 us; error 45 us; skew 500 ppm
Aug 06 15:55:48 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031748, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:55:48.595535 31771 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:55:48.596637 31771 webserver.cc:460] Webserver started at http://127.13.223.62:44013/ using document root <none> and password file <none>
I20260806 15:55:48.597205 31771 fs_manager.cc:359] Metadata directory not provided
I20260806 15:55:48.597285 31771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:55:48.597506 31771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:55:48 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031748, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:55:48.652129 31771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "c8f8467f4d8a4503a81520fac8a708ca"
format_stamp: "Formatted at 2026-08-06 15:55:48 on dist-test-slave-rwrg"
server_key: "50d0b32c6e56e2d32539f8a83f69c0d6"
server_key_iv: "6763f265d2568636c247c14df924ba9a"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:48.652901 31771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "c8f8467f4d8a4503a81520fac8a708ca"
format_stamp: "Formatted at 2026-08-06 15:55:48 on dist-test-slave-rwrg"
server_key: "50d0b32c6e56e2d32539f8a83f69c0d6"
server_key_iv: "6763f265d2568636c247c14df924ba9a"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:48.657446 31771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:55:48.672819 31795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:48.674398 31771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:55:48.674576 31771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "c8f8467f4d8a4503a81520fac8a708ca"
format_stamp: "Formatted at 2026-08-06 15:55:48 on dist-test-slave-rwrg"
server_key: "50d0b32c6e56e2d32539f8a83f69c0d6"
server_key_iv: "6763f265d2568636c247c14df924ba9a"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:48.674736 31771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:55:48.697329 31771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:55:48.700801 31771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:55:48.701099 31771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:55:48.717597 31771 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:40213
I20260806 15:55:48.717666 31871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:40213 every 8 connection(s)
I20260806 15:55:48.718925 31771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:55:48.723258 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:48.724457 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 31771
I20260806 15:55:48.724635 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:55:48.724925 14204 external_mini_cluster.cc:1428] Setting key 7afa9906447cc8f90f13d2821543eafc
I20260806 15:55:48.733073 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca: Bootstrap starting.
Aug 06 15:55:48 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:55:48.736915 31872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:48.738925 31872 log.cc:824] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:48.741874 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca: No bootstrap required, opened a new log
I20260806 15:55:48.742326 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:48.726458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.13.223.62:40213)
0806 15:55:48.727051 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:48.727095 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:48.727134 (+ 39us) server_negotiation.cc:407] Connection header received
0806 15:55:48.728062 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:48.728083 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:48.728434 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:48.728850 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:48.729854 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:48.731060 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:48.731927 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:48.732473 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:48.732711 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:48.735516 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:48.735522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:48.735536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:48.735567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:48.738862 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:48.739414 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:48.739422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:48.739434 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:48.739584 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:48.740001 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:48.740004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:48.740008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:48.740554 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:48.740764 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:48.741099 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 15:55:48.741420 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":242,"threads_started":1}
I20260806 15:55:48.747233 31872 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f8467f4d8a4503a81520fac8a708ca" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40213 } }
I20260806 15:55:48.747565 31872 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:48.747658 31872 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f8467f4d8a4503a81520fac8a708ca, State: Initialized, Role: FOLLOWER
I20260806 15:55:48.748183 31872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f8467f4d8a4503a81520fac8a708ca" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40213 } }
I20260806 15:55:48.748359 31872 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:55:48.748433 31872 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:55:48.748642 31872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:48.750470 31872 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f8467f4d8a4503a81520fac8a708ca" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40213 } }
I20260806 15:55:48.750876 31872 leader_election.cc:302] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f8467f4d8a4503a81520fac8a708ca; no voters:
I20260806 15:55:48.751348 31872 leader_election.cc:288] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:55:48.751664 31883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:55:48.752082 31883 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [term 1 LEADER]: Becoming Leader. State: Replica: c8f8467f4d8a4503a81520fac8a708ca, State: Running, Role: LEADER
I20260806 15:55:48.752576 31883 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f8467f4d8a4503a81520fac8a708ca" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40213 } }
I20260806 15:55:48.752821 31872 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:55:48.755059 31884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f8467f4d8a4503a81520fac8a708ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f8467f4d8a4503a81520fac8a708ca" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40213 } } }
I20260806 15:55:48.755059 31885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f8467f4d8a4503a81520fac8a708ca. Latest consensus state: current_term: 1 leader_uuid: "c8f8467f4d8a4503a81520fac8a708ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f8467f4d8a4503a81520fac8a708ca" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40213 } } }
I20260806 15:55:48.755203 31884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [sys.catalog]: This master's current role is: LEADER
I20260806 15:55:48.755311 31885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca [sys.catalog]: This master's current role is: LEADER
I20260806 15:55:48.756083 31893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:55:48.760627 31893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:55:48.761231 31872 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 15:55:48.769181 31893 catalog_manager.cc:1347] Generated new cluster ID: 55dff40c13c04022b0e2cb4a53a16259
I20260806 15:55:48.769281 31893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:55:48.807663 31893 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:55:48.809291 31893 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:55:48.820423 31893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca: Generated new TSK 0
I20260806 15:55:48.821606 31893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:55:49.791812 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:36745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40213
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:55:50.038038 31940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:55:50.038476 31940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:55:50.038668 31940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:55:50.043814 31940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:55:50.043883 31940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:55:50.043946 31940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:55:50.049800 31940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.223.20:36745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:55:50.050897 31940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:55:50.052024 31940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:55:50.064602 31949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:50.064570 31952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:50.064937 31958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:50.065850 31940 server_base.cc:1034] running on GCE node
I20260806 15:55:50.066283 31940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:55:50.067091 31940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:55:50.068284 31940 hybrid_clock.cc:648] HybridClock initialized: now 1786031750068256 us; error 39 us; skew 500 ppm
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:55:50.072170 31940 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:55:50.073589 31940 webserver.cc:460] Webserver started at http://127.13.223.1:33411/ using document root <none> and password file <none>
I20260806 15:55:50.074215 31940 fs_manager.cc:359] Metadata directory not provided
I20260806 15:55:50.074280 31940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:55:50.074527 31940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:55:50.104156 31940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "f0e8f4fd3e494d6c8d8050df39f36849"
format_stamp: "Formatted at 2026-08-06 15:55:50 on dist-test-slave-rwrg"
server_key: "d46832d2b68afa70bfadc896991e966b"
server_key_iv: "3858fc97e9d7a59c8e775c8410cb685d"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:50.105854 31940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "f0e8f4fd3e494d6c8d8050df39f36849"
format_stamp: "Formatted at 2026-08-06 15:55:50 on dist-test-slave-rwrg"
server_key: "d46832d2b68afa70bfadc896991e966b"
server_key_iv: "3858fc97e9d7a59c8e775c8410cb685d"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:50.114560 31940 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 15:55:50.133497 31968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:50.135215 31940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:55:50.135339 31940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "f0e8f4fd3e494d6c8d8050df39f36849"
format_stamp: "Formatted at 2026-08-06 15:55:50 on dist-test-slave-rwrg"
server_key: "d46832d2b68afa70bfadc896991e966b"
server_key_iv: "3858fc97e9d7a59c8e775c8410cb685d"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:50.135462 31940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:55:50.161193 31940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:55:50.164608 31940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:55:50.164841 31940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:55:50.165568 31940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:55:50.166585 31940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:55:50.166653 31940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:50.166751 31940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:55:50.166808 31940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:50.191833 31940 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:35177
I20260806 15:55:50.193032 32163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:35177 every 8 connection(s)
I20260806 15:55:50.193205 31940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:55:50.197237 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 31940
I20260806 15:55:50.197398 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:55:50.197763 14204 external_mini_cluster.cc:1428] Setting key fe4218f89ca0d05a9587e2bcb334bc41
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:55:50.233104 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:50.209444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:57631 (local address 127.13.223.62:40213)
0806 15:55:50.214470 (+ 5026us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:50.214476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:50.218114 (+ 3638us) server_negotiation.cc:407] Connection header received
0806 15:55:50.219655 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:50.219659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:50.219720 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:50.219833 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:50.221940 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.222862 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.224746 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.225170 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.225292 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:50.228236 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:50.228243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:50.228249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:50.228286 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:50.230273 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:50.230872 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:50.230875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:50.230877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:50.230961 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:50.231450 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:50.231456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:50.231460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:50.231758 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:50.231884 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:50.232680 (+ 796us) server_negotiation.cc:299] Negotiation successful
0806 15:55:50.232870 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4892,"thread_start_us":72,"threads_started":1}
I20260806 15:55:50.235522 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:50.210016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40213 (local address 127.13.223.1:57631)
0806 15:55:50.217892 (+ 7876us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:50.217955 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:50.219329 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:50.220173 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:50.220193 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:50.220848 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:55:50.221723 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:50.221745 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.223033 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.223036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:50.224590 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:50.224602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.225364 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.225369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:50.225586 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:50.226298 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:55:50.226319 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:55:50.228029 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:55:50.230396 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:50.230403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:50.230421 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:50.230751 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:50.231064 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:50.231070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:50.231075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:50.231277 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:50.231896 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:55:50.231906 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:55:50.232283 (+ 377us) client_negotiation.cc:769] Sending connection context
0806 15:55:50.233698 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0806 15:55:50.234142 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7411,"thread_start_us":245,"threads_started":1}
I20260806 15:55:50.237452 32164 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40213
I20260806 15:55:50.237879 32164 heartbeater.cc:464] Registering TS with master...
I20260806 15:55:50.238554 32164 heartbeater.cc:511] Master 127.13.223.62:40213 requested a full tablet report, sending...
I20260806 15:55:50.240449 31819 ts_manager.cc:194] Registered new tserver with Master: f0e8f4fd3e494d6c8d8050df39f36849 (127.13.223.1:35177)
I20260806 15:55:50.242434 31819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:57631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:55:50.251788 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:36745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40213
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 15:55:50.491874 32177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:55:50.492372 32177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:55:50.492533 32177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:55:50.499830 32177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:55:50.499920 32177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:55:50.500010 32177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:55:50.509223 32177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.223.20:36745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:55:50.510886 32177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:55:50.512611 32177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:55:50.527568 32187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:50.527612 32190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:50.527565 32188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:50.528779 32177 server_base.cc:1034] running on GCE node
I20260806 15:55:50.529330 32177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:55:50.530231 32177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:55:50.531451 32177 hybrid_clock.cc:648] HybridClock initialized: now 1786031750531436 us; error 32 us; skew 500 ppm
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:55:50.535946 32177 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:55:50.536952 32177 webserver.cc:460] Webserver started at http://127.13.223.2:40265/ using document root <none> and password file <none>
I20260806 15:55:50.537528 32177 fs_manager.cc:359] Metadata directory not provided
I20260806 15:55:50.537586 32177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:55:50.537815 32177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:55:50.556833 32177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "f350d224ea12433e947fb891ced371e8"
format_stamp: "Formatted at 2026-08-06 15:55:50 on dist-test-slave-rwrg"
server_key: "666b0f5761553de83d50fd7e9f5f0876"
server_key_iv: "c4318ec24a5425723e72d705b99ed935"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:50.557677 32177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "f350d224ea12433e947fb891ced371e8"
format_stamp: "Formatted at 2026-08-06 15:55:50 on dist-test-slave-rwrg"
server_key: "666b0f5761553de83d50fd7e9f5f0876"
server_key_iv: "c4318ec24a5425723e72d705b99ed935"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:50.563465 32177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 15:55:50.582000 32203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:50.583781 32177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 15:55:50.583915 32177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "f350d224ea12433e947fb891ced371e8"
format_stamp: "Formatted at 2026-08-06 15:55:50 on dist-test-slave-rwrg"
server_key: "666b0f5761553de83d50fd7e9f5f0876"
server_key_iv: "c4318ec24a5425723e72d705b99ed935"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:50.584040 32177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:55:50.650164 32177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:55:50.654515 32177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:55:50.654942 32177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:55:50.655853 32177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:55:50.657151 32177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:55:50.657246 32177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:50.657330 32177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:55:50.657378 32177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:50.679996 32177 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:44603
I20260806 15:55:50.681277 32177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:55:50.682422 32393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:44603 every 8 connection(s)
I20260806 15:55:50.688757 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 32177
I20260806 15:55:50.688908 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:55:50.689231 14204 external_mini_cluster.cc:1428] Setting key 4c41257d4b7f17c2177ad754b575225c
Aug 06 15:55:50 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:55:50.706857 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:50.687055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:45181 (local address 127.13.223.62:40213)
0806 15:55:50.687204 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:50.687209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:50.687227 (+ 18us) server_negotiation.cc:407] Connection header received
0806 15:55:50.688191 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:50.688195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:50.688302 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:50.688441 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:50.693155 (+ 4714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.694138 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.696479 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.696950 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.697207 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:50.702109 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:50.702114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:50.702116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:50.702160 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:50.704373 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:50.705107 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:50.705111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:50.705113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:50.705203 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:50.705634 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:50.705637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:50.705639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:50.705842 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:50.705955 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:50.706541 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 15:55:50.706681 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:55:50.707880 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:50.685762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40213 (local address 127.13.223.2:45181)
0806 15:55:50.686445 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:50.686500 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:50.687897 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:50.689895 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:50.689905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:50.690472 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:55:50.692876 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:50.692916 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.694346 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.694350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:50.696277 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:50.696292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:50.697447 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:50.697451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:50.697784 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:50.698769 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:55:50.698803 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:55:50.701806 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:55:50.704564 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:50.704570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:50.704588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:50.704933 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:50.705309 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:50.705312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:50.705314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:50.705519 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:50.705954 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:55:50.705963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:55:50.706287 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 15:55:50.706485 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 15:55:50.706829 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":211,"threads_started":1}
I20260806 15:55:50.709436 32394 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40213
I20260806 15:55:50.709731 32394 heartbeater.cc:464] Registering TS with master...
I20260806 15:55:50.710414 32394 heartbeater.cc:511] Master 127.13.223.62:40213 requested a full tablet report, sending...
I20260806 15:55:50.711766 31819 ts_manager.cc:194] Registered new tserver with Master: f350d224ea12433e947fb891ced371e8 (127.13.223.2:44603)
I20260806 15:55:50.712889 31819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:45181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:55:50.736491 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:36745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40213
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:55:50.970870 32403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:55:50.971199 32403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:55:50.971321 32403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:55:50.977926 32403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:55:50.978101 32403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:55:50.978246 32403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:55:50.986248 32403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.223.20:36745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:55:50.987887 32403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:55:50.989455 32403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:55:51.003347 32418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:55:51.004029 32420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:51.004056 32403 server_base.cc:1034] running on GCE node
W20260806 15:55:51.004124 32415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:55:51.004827 32403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:55:51.005555 32403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:55:51.013012 32403 hybrid_clock.cc:648] HybridClock initialized: now 1786031751012935 us; error 108 us; skew 500 ppm
Aug 06 15:55:51 dist-test-slave-rwrg krb5kdc[31069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031751, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:55:51.017788 32403 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:55:51.019337 32403 webserver.cc:460] Webserver started at http://127.13.223.3:41675/ using document root <none> and password file <none>
I20260806 15:55:51.020186 32403 fs_manager.cc:359] Metadata directory not provided
I20260806 15:55:51.020288 32403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:55:51.020704 32403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:55:51 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031751, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:55:51.040481 32403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "435d87f280434406a387dd1fd0b92b57"
format_stamp: "Formatted at 2026-08-06 15:55:51 on dist-test-slave-rwrg"
server_key: "4f4dd92fc04eec7903b4f5c6cc9023c9"
server_key_iv: "665933d0bc0a16beeff920301432d0f6"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:51.042014 32403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "435d87f280434406a387dd1fd0b92b57"
format_stamp: "Formatted at 2026-08-06 15:55:51 on dist-test-slave-rwrg"
server_key: "4f4dd92fc04eec7903b4f5c6cc9023c9"
server_key_iv: "665933d0bc0a16beeff920301432d0f6"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:51.071419 32403 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.005s sys 0.000s
Aug 06 15:55:51 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:55:51.096946 32429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:51.101598 32403 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260806 15:55:51.101778 32403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "435d87f280434406a387dd1fd0b92b57"
format_stamp: "Formatted at 2026-08-06 15:55:51 on dist-test-slave-rwrg"
server_key: "4f4dd92fc04eec7903b4f5c6cc9023c9"
server_key_iv: "665933d0bc0a16beeff920301432d0f6"
server_key_version: "kuduclusterkey@0"
I20260806 15:55:51.101969 32403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:55:51.135527 32403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:55:51.139786 32403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:55:51.140007 32403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:55:51.140961 32403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:55:51.142303 32403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:55:51.142421 32403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:51.142526 32403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:55:51.142598 32403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:55:51.158768 32403 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:43821
I20260806 15:55:51.160102 32403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:55:51.168135 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 32403
I20260806 15:55:51.168265 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:55:51.168598 14204 external_mini_cluster.cc:1428] Setting key 6567f305ea64c653299edfece6ba09e3
I20260806 15:55:51.173718 32614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:43821 every 8 connection(s)
Aug 06 15:55:51 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031751, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:55:51.190544 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.175858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:50715 (local address 127.13.223.62:40213)
0806 15:55:51.176026 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.176031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.177243 (+ 1212us) server_negotiation.cc:407] Connection header received
0806 15:55:51.178398 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.178402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.178463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.178550 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:51.180181 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.180844 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.182703 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.183047 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.183157 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.186275 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:51.186279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:51.186282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:51.186309 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:51.187995 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:51.188559 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:51.188567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:51.188568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:51.188647 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:51.189079 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:51.189083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:51.189085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:51.189342 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:51.189458 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.190212 (+ 754us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.190353 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:55:51.191599 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.176355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40213 (local address 127.13.223.3:50715)
0806 15:55:51.177061 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.177120 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:51.178169 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:51.178775 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:51.178793 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:51.179230 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:55:51.179970 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.179988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.180986 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.180990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.182567 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.182578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.183499 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.183502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.183692 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.184449 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:55:51.184467 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:55:51.186096 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:55:51.188125 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:51.188131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:51.188148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:51.188413 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:51.188780 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:55:51.188785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:55:51.188787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:55:51.188947 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:55:51.189536 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:55:51.189543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:55:51.189893 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 15:55:51.190144 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 15:55:51.190497 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
I20260806 15:55:51.197700 32615 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40213
I20260806 15:55:51.197979 32615 heartbeater.cc:464] Registering TS with master...
I20260806 15:55:51.198654 32615 heartbeater.cc:511] Master 127.13.223.62:40213 requested a full tablet report, sending...
I20260806 15:55:51.199855 31819 ts_manager.cc:194] Registered new tserver with Master: 435d87f280434406a387dd1fd0b92b57 (127.13.223.3:43821)
I20260806 15:55:51.200688 31819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:50715
I20260806 15:55:51.211074 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:55:51.222756 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.214104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.13.223.62:40213)
0806 15:55:51.214261 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.214266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.214474 (+ 208us) server_negotiation.cc:407] Connection header received
0806 15:55:51.214563 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.214566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.214625 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.214729 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:55:51.215430 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.216388 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.217296 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.217700 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.217819 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.219267 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:55:51.219274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:55:51.219277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:55:51.219309 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:55:51.221091 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:51.221502 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:51.221506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:51.221508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:51.221611 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:55:51.221879 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:55:51.221882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:55:51.221884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:55:51.222124 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:55:51.222266 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.222472 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.222599 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 15:55:51.226254 31819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:55:51.229485 31819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:55:51.245950 32164 heartbeater.cc:503] Master 127.13.223.62:40213 was elected leader, sending a full tablet report...
I20260806 15:55:51.248409 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.240671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:43821 (local address 127.0.0.1:42684)
0806 15:55:51.242263 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.242274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:51.242383 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:51.242978 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:51.242981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:51.242999 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:51.243098 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.243105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.244955 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.244958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.246413 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.246424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.247779 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.247783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.247885 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.247927 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 15:55:51.248009 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 15:55:51.248204 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":120,"thread_start_us":97,"threads_started":1}
I20260806 15:55:51.249861 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.240549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.13.223.1:35177)
0806 15:55:51.243186 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.243194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.243216 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:55:51.243286 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.243290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.243481 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.243686 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:51.244097 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.245292 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.247483 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.248275 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.248369 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.248536 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.249129 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.249276 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":175,"threads_started":1}
I20260806 15:55:51.252599 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.239864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.13.223.2:44603)
0806 15:55:51.241576 (+ 1712us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.241583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.241602 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:55:51.241720 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.241725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.241889 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.242012 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:51.243191 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.245240 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.250507 (+ 5267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.251358 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.251475 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.251628 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.252228 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.252337 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1615,"thread_start_us":189,"threads_started":1}
I20260806 15:55:51.257544 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.241120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.13.223.3:43821)
0806 15:55:51.241735 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.241769 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.242366 (+ 597us) server_negotiation.cc:407] Connection header received
0806 15:55:51.242498 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.242504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.242703 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.243250 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:51.243400 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.244815 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.246697 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.247646 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.256992 (+ 9346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.257195 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.257259 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.257373 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":193,"threads_started":1}
I20260806 15:55:51.248991 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.239557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:35177 (local address 127.0.0.1:38458)
0806 15:55:51.240660 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.240709 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:51.240837 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:51.243847 (+ 3010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:51.243850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:51.243867 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:51.243957 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.243963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.245420 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.245423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.247339 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.247350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.248598 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.248602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.248687 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.248708 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:55:51.248774 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 15:55:51.248853 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":699,"thread_start_us":113,"threads_started":1}
I20260806 15:55:51.261521 32514 tablet_service.cc:1467] Processing CreateTablet for tablet e55d14b5cd494c81a3daf781e79ff6cb (DEFAULT_TABLE table=test-table [id=71bef5e1569842f99275e55e83d70578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:55:51.263950 32514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55d14b5cd494c81a3daf781e79ff6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:51.263805 32045 tablet_service.cc:1467] Processing CreateTablet for tablet e55d14b5cd494c81a3daf781e79ff6cb (DEFAULT_TABLE table=test-table [id=71bef5e1569842f99275e55e83d70578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:55:51.264969 32045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55d14b5cd494c81a3daf781e79ff6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:51.252125 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.239638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44603 (local address 127.0.0.1:39296)
0806 15:55:51.241197 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.241212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:51.241327 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:51.242135 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:51.242143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:51.242676 (+ 533us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:51.242924 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.242946 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.245471 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.245484 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.250339 (+ 4855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.250352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.251702 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.251707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.251803 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.251826 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:55:51.251903 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 15:55:51.251972 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"thread_start_us":84,"threads_started":1}
I20260806 15:55:51.271885 32279 tablet_service.cc:1467] Processing CreateTablet for tablet e55d14b5cd494c81a3daf781e79ff6cb (DEFAULT_TABLE table=test-table [id=71bef5e1569842f99275e55e83d70578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:55:51.273027 32279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55d14b5cd494c81a3daf781e79ff6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:55:51.274338 32643 tablet_bootstrap.cc:492] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Bootstrap starting.
I20260806 15:55:51.277870 32643 tablet_bootstrap.cc:654] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:51.279296 32643 log.cc:824] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:51.281782 32643 tablet_bootstrap.cc:492] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: No bootstrap required, opened a new log
I20260806 15:55:51.282001 32643 ts_tablet_manager.cc:1397] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 15:55:51.286686 32643 raft_consensus.cc:348] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.286953 32643 raft_consensus.cc:374] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:51.287022 32643 raft_consensus.cc:729] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435d87f280434406a387dd1fd0b92b57, State: Initialized, Role: FOLLOWER
I20260806 15:55:51.287559 32643 consensus_queue.cc:260] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.287624 32645 tablet_bootstrap.cc:492] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: Bootstrap starting.
I20260806 15:55:51.288653 32643 ts_tablet_manager.cc:1428] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 15:55:51.289063 32615 heartbeater.cc:503] Master 127.13.223.62:40213 was elected leader, sending a full tablet report...
I20260806 15:55:51.290699 32647 tablet_bootstrap.cc:492] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: Bootstrap starting.
I20260806 15:55:51.294039 32647 tablet_bootstrap.cc:654] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:51.294773 32645 tablet_bootstrap.cc:654] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: Neither blocks nor log segments found. Creating new log.
I20260806 15:55:51.295228 32647 log.cc:824] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:51.295800 32645 log.cc:824] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: Log is configured to *not* fsync() on all Append() calls
I20260806 15:55:51.297575 32647 tablet_bootstrap.cc:492] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: No bootstrap required, opened a new log
I20260806 15:55:51.297762 32647 ts_tablet_manager.cc:1397] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 15:55:51.297943 32645 tablet_bootstrap.cc:492] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: No bootstrap required, opened a new log
I20260806 15:55:51.298121 32645 ts_tablet_manager.cc:1397] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260806 15:55:51.303186 32645 raft_consensus.cc:348] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.303485 32645 raft_consensus.cc:374] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:51.303555 32645 raft_consensus.cc:729] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f350d224ea12433e947fb891ced371e8, State: Initialized, Role: FOLLOWER
I20260806 15:55:51.303918 32647 raft_consensus.cc:348] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.304088 32645 consensus_queue.cc:260] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.304205 32647 raft_consensus.cc:374] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:55:51.304435 32647 raft_consensus.cc:729] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e8f4fd3e494d6c8d8050df39f36849, State: Initialized, Role: FOLLOWER
I20260806 15:55:51.304947 32647 consensus_queue.cc:260] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.305322 32645 ts_tablet_manager.cc:1428] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260806 15:55:51.305727 32394 heartbeater.cc:503] Master 127.13.223.62:40213 was elected leader, sending a full tablet report...
I20260806 15:55:51.306149 32647 ts_tablet_manager.cc:1428] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
Aug 06 15:55:51 dist-test-slave-rwrg krb5kdc[31069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
W20260806 15:55:51.426040 32616 tablet.cc:2369] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:55:51.433728 32396 tablet.cc:2369] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:55:51.459136 32166 tablet.cc:2369] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:55:51.508653 32655 raft_consensus.cc:482] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:55:51.508931 32655 raft_consensus.cc:504] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.510214 32655 leader_election.cc:288] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f350d224ea12433e947fb891ced371e8 (127.13.223.2:44603), 435d87f280434406a387dd1fd0b92b57 (127.13.223.3:43821)
I20260806 15:55:51.522440 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.510993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:41619 (local address 127.13.223.3:43821)
0806 15:55:51.511130 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.511135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.517813 (+ 6678us) server_negotiation.cc:407] Connection header received
0806 15:55:51.517881 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.517884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.517942 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.518023 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:51.518495 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.519287 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.521045 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.521667 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.521768 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.521820 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.522217 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.522301 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:55:51.523182 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.510495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:43821 (local address 127.13.223.1:41619)
0806 15:55:51.517519 (+ 7024us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.517537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:51.517677 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:51.518140 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:51.518143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:51.518161 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:51.518330 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.518337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.519503 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.519506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.520890 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.520901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.521923 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.521927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.522055 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.522079 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:55:51.522872 (+ 793us) client_negotiation.cc:240] Negotiation successful
0806 15:55:51.522955 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6920,"thread_start_us":113,"threads_started":1}
I20260806 15:55:51.524080 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.511005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:51339 (local address 127.13.223.2:44603)
0806 15:55:51.511248 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.511253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.512878 (+ 1625us) server_negotiation.cc:407] Connection header received
0806 15:55:51.513104 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.513107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.513164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.513231 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:55:51.513830 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.514622 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.516366 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.517036 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.517138 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.517208 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.523874 (+ 6666us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.523966 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260806 15:55:51.524091 32544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55d14b5cd494c81a3daf781e79ff6cb" candidate_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435d87f280434406a387dd1fd0b92b57" is_pre_election: true
I20260806 15:55:51.524456 32544 raft_consensus.cc:2454] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e8f4fd3e494d6c8d8050df39f36849 in term 0.
I20260806 15:55:51.524741 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.510458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44603 (local address 127.13.223.1:51339)
0806 15:55:51.512767 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.512785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:55:51.512970 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:55:51.513421 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:55:51.513425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:55:51.513445 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:55:51.513682 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.513691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.514780 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.514784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.516214 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:55:51.516225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.517366 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.517370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:55:51.517489 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.523519 (+ 6030us) client_negotiation.cc:769] Sending connection context
0806 15:55:51.524394 (+ 875us) client_negotiation.cc:240] Negotiation successful
0806 15:55:51.524479 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"thread_start_us":147,"threads_started":1}
I20260806 15:55:51.525199 31971 leader_election.cc:302] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435d87f280434406a387dd1fd0b92b57, f0e8f4fd3e494d6c8d8050df39f36849; no voters:
I20260806 15:55:51.525514 32655 raft_consensus.cc:2781] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:55:51.525605 32655 raft_consensus.cc:482] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:55:51.525652 32655 raft_consensus.cc:3037] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:51.525992 32305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55d14b5cd494c81a3daf781e79ff6cb" candidate_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f350d224ea12433e947fb891ced371e8" is_pre_election: true
I20260806 15:55:51.526268 32305 raft_consensus.cc:2454] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e8f4fd3e494d6c8d8050df39f36849 in term 0.
I20260806 15:55:51.527462 32655 raft_consensus.cc:504] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.527841 32655 leader_election.cc:288] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [CANDIDATE]: Term 1 election: Requested vote from peers f350d224ea12433e947fb891ced371e8 (127.13.223.2:44603), 435d87f280434406a387dd1fd0b92b57 (127.13.223.3:43821)
I20260806 15:55:51.528292 32305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55d14b5cd494c81a3daf781e79ff6cb" candidate_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f350d224ea12433e947fb891ced371e8"
I20260806 15:55:51.528421 32305 raft_consensus.cc:3037] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:51.529510 32305 raft_consensus.cc:2454] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e8f4fd3e494d6c8d8050df39f36849 in term 1.
I20260806 15:55:51.529811 32544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55d14b5cd494c81a3daf781e79ff6cb" candidate_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435d87f280434406a387dd1fd0b92b57"
I20260806 15:55:51.529879 31969 leader_election.cc:302] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e8f4fd3e494d6c8d8050df39f36849, f350d224ea12433e947fb891ced371e8; no voters:
I20260806 15:55:51.529944 32544 raft_consensus.cc:3037] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:55:51.530133 32655 raft_consensus.cc:2781] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:55:51.530413 32655 raft_consensus.cc:686] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [term 1 LEADER]: Becoming Leader. State: Replica: f0e8f4fd3e494d6c8d8050df39f36849, State: Running, Role: LEADER
I20260806 15:55:51.530759 32655 consensus_queue.cc:237] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } }
I20260806 15:55:51.531294 32544 raft_consensus.cc:2454] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e8f4fd3e494d6c8d8050df39f36849 in term 1.
I20260806 15:55:51.534430 31817 catalog_manager.cc:5495] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e8f4fd3e494d6c8d8050df39f36849 (127.13.223.1). New cstate: current_term: 1 leader_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e8f4fd3e494d6c8d8050df39f36849" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 35177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 } health_report { overall_health: UNKNOWN } } }
I20260806 15:55:51.614609 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.609497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.13.223.1:35177)
0806 15:55:51.609624 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 15:55:51.609629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:55:51.609654 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:55:51.609748 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:55:51.609752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:55:51.609846 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:55:51.609968 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:55:51.610368 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.611157 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.612528 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:55:51.612874 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:55:51.613010 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:55:51.613446 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:55:51.614069 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:55:51.614220 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 15:55:51.614327 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 15:55:51.614400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:55:51.622459 32305 raft_consensus.cc:1261] T e55d14b5cd494c81a3daf781e79ff6cb P f350d224ea12433e947fb891ced371e8 [term 1 FOLLOWER]: Refusing update from remote peer f0e8f4fd3e494d6c8d8050df39f36849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:55:51.622459 32544 raft_consensus.cc:1261] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 1 FOLLOWER]: Refusing update from remote peer f0e8f4fd3e494d6c8d8050df39f36849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:55:51.623462 32655 consensus_queue.cc:1035] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "435d87f280434406a387dd1fd0b92b57" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:55:51.624125 32655 consensus_queue.cc:1035] T e55d14b5cd494c81a3daf781e79ff6cb P f0e8f4fd3e494d6c8d8050df39f36849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f350d224ea12433e947fb891ced371e8" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:55:51.634164 32667 mvcc.cc:205] Tried to move back new op lower bound from 7315586054637207552 to 7315586054273871872. Current Snapshot: MvccSnapshot[applied={T|T < 7315586054637207552}]
I20260806 15:55:51.634696 32668 mvcc.cc:205] Tried to move back new op lower bound from 7315586054637207552 to 7315586054273871872. Current Snapshot: MvccSnapshot[applied={T|T < 7315586054637207552}]
I20260806 15:55:51.638720 32669 mvcc.cc:205] Tried to move back new op lower bound from 7315586054637207552 to 7315586054273871872. Current Snapshot: MvccSnapshot[applied={T|T < 7315586054637207552}]
I20260806 15:55:51.651316 31819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:55:51.651520 31819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:55:51.654698 31819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8f8467f4d8a4503a81520fac8a708ca: Sending DeleteTablet for 3 replicas of tablet e55d14b5cd494c81a3daf781e79ff6cb
I20260806 15:55:51.656100 32045 tablet_service.cc:1514] Processing DeleteTablet for tablet e55d14b5cd494c81a3daf781e79ff6cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:55:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:38458
I20260806 15:55:51.656976 32279 tablet_service.cc:1514] Processing DeleteTablet for tablet e55d14b5cd494c81a3daf781e79ff6cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:55:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:39296
I20260806 15:55:51.657392 32514 tablet_service.cc:1514] Processing DeleteTablet for tablet e55d14b5cd494c81a3daf781e79ff6cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:55:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:42684
I20260806 15:55:51.658672 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 31940
I20260806 15:55:51.672986 32683 tablet_replica.cc:330] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: stopping tablet replica
I20260806 15:55:51.674490 32683 raft_consensus.cc:2229] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:55:51.675630 32683 raft_consensus.cc:2258] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 15:55:51.679461 31800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:35177 (error 108)
I20260806 15:55:51.680456 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 32177
I20260806 15:55:51.685385 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.684999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:35177 (local address 127.0.0.1:38474)
0806 15:55:51.685212 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.685290 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:35177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260806 15:55:51.685720 31800 catalog_manager.cc:4584] TS f0e8f4fd3e494d6c8d8050df39f36849 (127.13.223.1:35177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e55d14b5cd494c81a3daf781e79ff6cb: Network error: Client connection negotiation failed: client connection to 127.13.223.1:35177: connect: Connection refused (error 111)
I20260806 15:55:51.690690 32683 ts_tablet_manager.cc:1905] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:55:51.694785 32683 ts_tablet_manager.cc:1918] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:55:51.694972 32683 log.cc:1196] T e55d14b5cd494c81a3daf781e79ff6cb P 435d87f280434406a387dd1fd0b92b57: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/e55d14b5cd494c81a3daf781e79ff6cb
I20260806 15:55:51.695673 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 32403
I20260806 15:55:51.706689 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.706130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44603 (local address 127.0.0.1:39304)
0806 15:55:51.706377 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.706502 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.2:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 15:55:51.728443 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 31771
I20260806 15:55:51.728609 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:55:51.728265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:43821 (local address 127.0.0.1:42700)
0806 15:55:51.728449 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 15:55:51.728520 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.3:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 15:55:51.740836 14204 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 15:55:52.089205 14204 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 15:55:52.089386 14204 mini_ranger.cc:66] Stopping Ranger...
I20260806 15:55:52.435194 14204 mini_ranger.cc:68] Stopped Ranger
2026-08-06 15:55:52.435 UTC [31115] LOG: received smart shutdown request
2026-08-06 15:55:52.441 UTC [31115] LOG: background worker "logical replication launcher" (PID 31131) exited with exit code 1
2026-08-06 15:55:52.441 UTC [31120] LOG: shutting down
2026-08-06 15:55:52.612 UTC [31115] LOG: database system is shut down
2026-08-06T15:55:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52259 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:55:52 dist-test-slave-rwrg krb5kdc[32714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:55:52 dist-test-slave-rwrg krb5kdc[32714](info): set up 2 sockets
Aug 06 15:55:52 dist-test-slave-rwrg krb5kdc[32714](info): commencing operation
krb5kdc: starting...
W20260806 15:55:54.038760 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:55:54 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:55:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/rangeradmin_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/rangeradmin_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/rangerlookup_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/rangerlookup_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
I20260806 15:55:54.225327 14204 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBrJzxO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/postgres -l logfile start
2026-08-06 15:55:56.713 UTC [300] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 15:55:56.713 UTC [300] LOG: listening on IPv4 address "127.13.223.20", port 43815
2026-08-06 15:55:56.717 UTC [300] LOG: listening on Unix socket "/tmp/.s.PGSQL.43815"
2026-08-06 15:55:56.738 UTC [305] LOG: database system was shut down at 2026-08-06 15:55:55 UTC
2026-08-06 15:55:56.742 UTC [300] LOG: database system is ready to accept connections
I20260806 15:55:57.894567 14204 mini_postgres.cc:96] Postgres bound to 43815
2026-08-06 15:55:57.902 UTC [315] FATAL: database "slave" does not exist
127.13.223.20:43815 - accepting connections
I20260806 15:55:57.902669 14204 mini_ranger.cc:161] Starting Ranger...
I20260806 15:55:57.919091 14204 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 15:55:58.398350 14204 mini_ranger.cc:87] Created ranger Postgres database
I20260806 15:55:58.398586 14204 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-admin
2026-08-06 15:55:58,663 [I] DB FLAVOR :POSTGRES
2026-08-06 15:55:58,663 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 15:55:58,663 [I] Checking connection..
2026-08-06 15:55:58,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 15:55:58,907 [I] Checking connection passed.
2026-08-06 15:55:58,907 [I] --------- Verifying version history table ---------
2026-08-06 15:55:58,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 15:55:59,199 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 15:55:59,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 15:55:59,459 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 15:55:59,459 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 15:55:59,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 15:55:59.712 UTC [423] WARNING: there is no transaction in progress
2026-08-06 15:55:59,722 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 15:55:59,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 15:55:59,984 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 15:55:59,984 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 15:55:59,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 15:56:00,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 15:56:00,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rwrg.c.gcp-upstream.internal','N') ;"
2026-08-06 15:56:00,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 15:56:00,989 [I] Table x_portal_user does not exist in database ranger
2026-08-06 15:56:00,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 15:56:01,232 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 15:56:01,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 15:56:01,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 15:56:01,760 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 15:56:01,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 15:56:02.775 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.791 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.808 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.826 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.847 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.868 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.928 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.936 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.951 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.963 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.974 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.987 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:02.997 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:03.004 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:03.011 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:03.018 UTC [591] WARNING: there is no transaction in progress
2026-08-06 15:56:03,706 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 15:56:03,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 15:56:03,962 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 15:56:03,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 15:56:04,212 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 15:56:04,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 15:56:04,471 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 15:56:04,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 15:56:04,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 15:56:04,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rwrg.c.gcp-upstream.internal';"
2026-08-06 15:56:05,230 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 15:56:05,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 15:56:05,480 [I] DB_PATCHES have already been applied
I20260806 15:56:05.486720 14204 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBrJzxO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 15:56:05.486788 14204 mini_ranger.cc:193] Using host: 127.13.223.20
I20260806 15:56:05.491197 14204 mini_ranger.cc:227] Ranger admin URL: http://127.13.223.20:45013
Aug 06, 2026 3:56:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:56:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:56:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 3:56:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 3:56:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 3:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 3:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 15:56:06.871217 14204 mini_ranger.cc:160] Time spent starting Ranger: real 8.969s user 0.000s sys 0.012s
Aug 06, 2026 3:56:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 3:56:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 3:56:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 3:56:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 3:56:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 15:56:28.964741 14204 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.223.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.223.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/rangerkms_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.223.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/rangerkms_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.223.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/HTTP_127.13.223.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 15:56:29 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031789, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 15:56:29.056803 14204 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 15:56:29.067216 14204 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 15:56:29.920676 14204 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 15:56:29.920769 14204 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms
I20260806 15:56:29.920789 14204 mini_ranger_kms.cc:227] Using postgres at 127.13.223.20:43815
2026-08-06 15:56:30,114 [I] DB FLAVOR :POSTGRES
2026-08-06 15:56:30,115 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 15:56:30,115 [I] Checking connection
2026-08-06 15:56:30,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 15:56:30,373 [I] connection success
2026-08-06 15:56:30,373 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 15:56:30,373 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 15:56:30,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 15:56:30,637 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 15:56:30,637 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 15:56:30,637 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 15:56:30,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.20:43815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 15:56:30,903 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 15:56:31.406811 14204 mini_ranger_kms.cc:310] Created kms service
I20260806 15:56:31.624428 14204 mini_ranger_kms.cc:326] Created kudu user
I20260806 15:56:31.669288 14204 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 15:56:31 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031789, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:32.103960 14204 mini_ranger_kms.cc:384] Added ranger policy
I20260806 15:56:32.104132 14204 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBrJzxO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBrJzxO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 15:56:32.104156 14204 mini_ranger_kms.cc:240] Using host: 127.13.223.20
I20260806 15:56:32.109463 14204 mini_ranger_kms.cc:276] Ranger KMS PID: 879
I20260806 15:56:32.109570 14204 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.223.20:46533
Aug 06, 2026 3:56:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:56:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:56:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 3:56:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 3:56:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 3:56:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 15:56:34 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786031794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 15:56:34 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.20@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:34.926852 14204 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.870s user 0.007s sys 0.164s
I20260806 15:56:34.926970 14204 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 15:56:34.927034 14204 mini_ranger_kms.cc:401] 127.13.223.20:46533/kms/v1/keys
Aug 06 15:56:35 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.20@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
Aug 06 15:56:37 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:56:37.491801 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34489
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34489
--ranger_config_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:37.609117 975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:37.609402 975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:37.609481 975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:37.613507 975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:37.613575 975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:37.613595 975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:37.613611 975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:37.613626 975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:37.619510 975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.223.20:46533/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34489
--ranger_config_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:37.620649 975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:37.621706 975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:56:37.631018 975 server_base.cc:1034] running on GCE node
W20260806 15:56:37.630973 986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:37.630973 983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:37.631007 984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:37.631954 975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:37.633139 975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:37.634307 975 hybrid_clock.cc:648] HybridClock initialized: now 1786031797634295 us; error 41 us; skew 500 ppm
Aug 06 15:56:37 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031797, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:37.641753 975 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:37.642696 975 webserver.cc:460] Webserver started at http://127.13.223.62:38759/ using document root <none> and password file <none>
I20260806 15:56:37.643239 975 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:37.643342 975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:37.643651 975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:56:37 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031797, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:37.701128 975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "47504d804b664f2e9afa765ab51c5fee"
format_stamp: "Formatted at 2026-08-06 15:56:37 on dist-test-slave-rwrg"
server_key: "0ccd034fe17fde819a55dfb171609cfe"
server_key_iv: "f81a44e2bdfb6004754075fb2723beb7"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:37.701889 975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "47504d804b664f2e9afa765ab51c5fee"
format_stamp: "Formatted at 2026-08-06 15:56:37 on dist-test-slave-rwrg"
server_key: "0ccd034fe17fde819a55dfb171609cfe"
server_key_iv: "f81a44e2bdfb6004754075fb2723beb7"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:37.707386 975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 15:56:37.731298 993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:37.732945 975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:56:37.733106 975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "47504d804b664f2e9afa765ab51c5fee"
format_stamp: "Formatted at 2026-08-06 15:56:37 on dist-test-slave-rwrg"
server_key: "0ccd034fe17fde819a55dfb171609cfe"
server_key_iv: "f81a44e2bdfb6004754075fb2723beb7"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:37.733260 975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:37.745862 975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:37.749101 975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:37.749358 975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:37.757268 975 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34489
I20260806 15:56:37.757277 1076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34489 every 8 connection(s)
I20260806 15:56:37.758342 975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:37.761173 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 975
I20260806 15:56:37.761278 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:56:37.761373 1078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:37.761634 14204 external_mini_cluster.cc:1428] Setting key 26e72965cb55f4abb07ff59b5b4ab6d4
I20260806 15:56:37.768451 1078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Bootstrap starting.
I20260806 15:56:37.771294 1078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Neither blocks nor log segments found. Creating new log.
Aug 06 15:56:37 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:37.772146 1078 log.cc:824] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:37.774192 1078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: No bootstrap required, opened a new log
I20260806 15:56:37.777310 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:37.763241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.13.223.62:34489)
0806 15:56:37.763680 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:37.763697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:37.763734 (+ 37us) server_negotiation.cc:407] Connection header received
0806 15:56:37.764721 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:37.764741 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:37.765090 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:37.765377 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:37.766434 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:37.767441 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:37.768624 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:37.769116 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:37.769271 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:37.772138 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:37.772143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:37.772157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:37.772185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:37.774593 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:37.775066 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:37.775071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:37.775076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:37.775180 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:37.775460 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:37.775464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:37.775465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:37.775842 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:37.776050 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:37.776333 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 15:56:37.776627 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260806 15:56:37.777202 1078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:37.777467 1078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:37.777537 1078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47504d804b664f2e9afa765ab51c5fee, State: Initialized, Role: FOLLOWER
I20260806 15:56:37.777987 1078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:37.778105 1078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:56:37.778146 1078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:56:37.778208 1078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:37.779310 1078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:37.779664 1078 leader_election.cc:302] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47504d804b664f2e9afa765ab51c5fee; no voters:
I20260806 15:56:37.779969 1078 leader_election.cc:288] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:56:37.780054 1088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:37.780330 1088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 LEADER]: Becoming Leader. State: Replica: 47504d804b664f2e9afa765ab51c5fee, State: Running, Role: LEADER
I20260806 15:56:37.780776 1088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:37.781229 1078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:37.782439 1089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47504d804b664f2e9afa765ab51c5fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } } }
I20260806 15:56:37.782585 1089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:37.782887 1090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47504d804b664f2e9afa765ab51c5fee. Latest consensus state: current_term: 1 leader_uuid: "47504d804b664f2e9afa765ab51c5fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } } }
I20260806 15:56:37.782995 1090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:37.782982 1098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:37.783898 1078 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 15:56:37.790690 1098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:37.798605 1098 catalog_manager.cc:1347] Generated new cluster ID: 53b11ec0dbc24cb89426d56496639772
I20260806 15:56:37.798676 1098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:56:37.841310 1098 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:56:37.842715 1098 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:56:37.856805 1098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Generated new TSK 0
I20260806 15:56:37.858407 1098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:56:38.599614 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34489
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:38.743613 1148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:38.744056 1148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:38.744153 1148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:38.750221 1148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:38.750375 1148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:38.750487 1148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:56:38.756942 1148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.223.20:46533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:38.758388 1148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:38.759670 1148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:38.771848 1171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:38.771848 1166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:38.772397 1148 server_base.cc:1034] running on GCE node
W20260806 15:56:38.772579 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:38.773102 1148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:38.773797 1148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:38.774997 1148 hybrid_clock.cc:648] HybridClock initialized: now 1786031798774971 us; error 47 us; skew 500 ppm
Aug 06 15:56:38 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031798, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:38.778164 1148 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:56:38.779098 1148 webserver.cc:460] Webserver started at http://127.13.223.1:35175/ using document root <none> and password file <none>
I20260806 15:56:38.779915 1148 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:38.780035 1148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:38.780359 1148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:56:38 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031798, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:38.802311 1148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "366a56dd746f4e8a9497879327eeb294"
format_stamp: "Formatted at 2026-08-06 15:56:38 on dist-test-slave-rwrg"
server_key: "a2ad4bf2f0f4e1e7ba79f488f4455057"
server_key_iv: "508dfa102c26d9dd86ed684a0f011055"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:38.803006 1148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "366a56dd746f4e8a9497879327eeb294"
format_stamp: "Formatted at 2026-08-06 15:56:38 on dist-test-slave-rwrg"
server_key: "a2ad4bf2f0f4e1e7ba79f488f4455057"
server_key_iv: "508dfa102c26d9dd86ed684a0f011055"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:38.807293 1148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:56:38.836747 1181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:38.847761 1148 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.002s sys 0.000s
I20260806 15:56:38.847951 1148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "366a56dd746f4e8a9497879327eeb294"
format_stamp: "Formatted at 2026-08-06 15:56:38 on dist-test-slave-rwrg"
server_key: "a2ad4bf2f0f4e1e7ba79f488f4455057"
server_key_iv: "508dfa102c26d9dd86ed684a0f011055"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:38.848124 1148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:38.878824 1148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:38.882984 1148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:38.883286 1148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:38.884162 1148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:38.885308 1148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:38.885421 1148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:38.885555 1148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:38.885658 1148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:38.898206 1148 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:46297
I20260806 15:56:38.899480 1148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:56:38.900091 1357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:46297 every 8 connection(s)
I20260806 15:56:38.903609 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 1148
I20260806 15:56:38.903730 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:56:38.904031 14204 external_mini_cluster.cc:1428] Setting key 888761d8dadecbcd9053dea2de6f7a7d
Aug 06 15:56:38 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031798, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:56:38.919262 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:38.902588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:34623 (local address 127.13.223.62:34489)
0806 15:56:38.902897 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:38.902902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:38.904606 (+ 1704us) server_negotiation.cc:407] Connection header received
0806 15:56:38.906329 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:38.906332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:38.906386 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:38.906471 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:38.908018 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:38.908915 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:38.910637 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:38.911012 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:38.911147 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:38.914388 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:38.914392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:38.914395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:38.914425 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:38.916440 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:38.917173 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:38.917177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:38.917179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:38.917276 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:38.917742 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:38.917746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:38.917748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:38.917996 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:38.918114 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:38.918912 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 15:56:38.919062 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260806 15:56:38.920384 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:38.903075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34489 (local address 127.13.223.1:34623)
0806 15:56:38.904283 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:38.904345 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:38.905990 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:38.906800 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:38.906811 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:38.907174 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:38.907848 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:38.907866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:38.909064 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:38.909068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:38.910493 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:38.910504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:38.911333 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:38.911338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:38.911606 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:38.912344 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:38.912366 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:38.914180 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:38.916621 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:38.916627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:38.916644 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:38.917017 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:38.917412 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:38.917415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:38.917417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:38.917620 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:38.918177 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:38.918185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:38.918560 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 15:56:38.918868 (+ 308us) client_negotiation.cc:240] Negotiation successful
0806 15:56:38.919275 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:56:38.922102 1358 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34489
I20260806 15:56:38.922354 1358 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:38.922968 1358 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:38.924858 1022 ts_manager.cc:194] Registered new tserver with Master: 366a56dd746f4e8a9497879327eeb294 (127.13.223.1:46297)
I20260806 15:56:38.926465 1022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:34623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:56:38.944080 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34489
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:56:38 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786031798, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 15:56:39.103498 1370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:39.103822 1370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:39.103910 1370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:39.109342 1370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:39.109426 1370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:39.109512 1370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:56:39.115316 1370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.223.20:46533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:39.116389 1370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:39.117475 1370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:39.128124 1384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:39.128257 1381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:39.128125 1382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:39.128978 1370 server_base.cc:1034] running on GCE node
I20260806 15:56:39.129410 1370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:39.130096 1370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:39.131340 1370 hybrid_clock.cc:648] HybridClock initialized: now 1786031799131316 us; error 39 us; skew 500 ppm
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:39.135253 1370 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:56:39.136498 1370 webserver.cc:460] Webserver started at http://127.13.223.2:34767/ using document root <none> and password file <none>
I20260806 15:56:39.137401 1370 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:39.137576 1370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:39.137890 1370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:39.157999 1370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "a2be3674eeb54f489147f8114006ee37"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "8acc844c70c9c7f265a9f6aeea8b1a39"
server_key_iv: "9436a0122a0817e68eef80e160ac8416"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.158764 1370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "a2be3674eeb54f489147f8114006ee37"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "8acc844c70c9c7f265a9f6aeea8b1a39"
server_key_iv: "9436a0122a0817e68eef80e160ac8416"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.164112 1370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 15:56:39.181717 1393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.183558 1370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 15:56:39.183730 1370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "a2be3674eeb54f489147f8114006ee37"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "8acc844c70c9c7f265a9f6aeea8b1a39"
server_key_iv: "9436a0122a0817e68eef80e160ac8416"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.183883 1370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:39.211436 1370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:39.214753 1370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:39.214967 1370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:39.215660 1370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:39.216709 1370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:39.216780 1370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.216850 1370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:39.216902 1370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.231642 1370 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:43969
I20260806 15:56:39.232870 1370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:56:39.235862 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 1370
I20260806 15:56:39.235999 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:56:39.236336 14204 external_mini_cluster.cc:1428] Setting key a0e6ae665ae3edd84f83dc84c0a13013
I20260806 15:56:39.241757 1589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:43969 every 8 connection(s)
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:39.262683 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:39.243767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:43287 (local address 127.13.223.62:34489)
0806 15:56:39.243918 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:39.243923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:39.245079 (+ 1156us) server_negotiation.cc:407] Connection header received
0806 15:56:39.246384 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:39.246389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:39.246444 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:39.246500 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:39.248215 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.249106 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.250863 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.251240 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.251320 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:39.257874 (+ 6554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:39.257878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:39.257881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:39.257914 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:39.259893 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:39.260584 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:39.260588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:39.260590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:39.260679 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:39.261107 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:39.261110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:39.261112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:39.261340 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:39.261437 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:39.262243 (+ 806us) server_negotiation.cc:299] Negotiation successful
0806 15:56:39.262400 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 15:56:39.264460 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:39.244300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34489 (local address 127.13.223.2:43287)
0806 15:56:39.244898 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:39.244963 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:39.246115 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:39.246936 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:39.246946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:39.247327 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:39.248030 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:39.248052 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.249291 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.249294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:39.250692 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:39.250704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.251534 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.251538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:39.251749 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:39.252483 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:39.252537 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:39.254858 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:39.260035 (+ 5177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:39.260042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:39.260062 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:39.260438 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:39.260807 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:39.260810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:39.260812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:39.260995 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:39.261516 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:39.261526 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:39.261907 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 15:56:39.262952 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0806 15:56:39.263308 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":166,"threads_started":1}
I20260806 15:56:39.265914 1590 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34489
I20260806 15:56:39.266191 1590 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:39.266805 1590 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:39.268062 1022 ts_manager.cc:194] Registered new tserver with Master: a2be3674eeb54f489147f8114006ee37 (127.13.223.2:43969)
I20260806 15:56:39.268859 1022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:43287
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:56:39.301998 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34489
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031798, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
W20260806 15:56:39.469101 1602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:39.469487 1602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:39.469630 1602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:39.475440 1602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:39.475524 1602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:39.475613 1602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:56:39.483065 1602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.223.20:46533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:39.484381 1602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:39.485687 1602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:39.496994 1612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:39.497143 1611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:39.497030 1621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:39.497430 1602 server_base.cc:1034] running on GCE node
I20260806 15:56:39.497859 1602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:39.498651 1602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:39.499833 1602 hybrid_clock.cc:648] HybridClock initialized: now 1786031799499801 us; error 45 us; skew 500 ppm
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:39.502928 1602 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:56:39.504004 1602 webserver.cc:460] Webserver started at http://127.13.223.3:43917/ using document root <none> and password file <none>
I20260806 15:56:39.504678 1602 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:39.504781 1602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:39.505000 1602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:39.521634 1602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "02337781433d4aaf4d615b6f197a8554"
server_key_iv: "6143c87adb146db36064b9eb7ba91484"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.522408 1602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "02337781433d4aaf4d615b6f197a8554"
server_key_iv: "6143c87adb146db36064b9eb7ba91484"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.550390 1602 fs_manager.cc:691] Time spent creating directory manager: real 0.028s user 0.006s sys 0.000s
I20260806 15:56:39.574554 1631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.576557 1602 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260806 15:56:39.576725 1602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "02337781433d4aaf4d615b6f197a8554"
server_key_iv: "6143c87adb146db36064b9eb7ba91484"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.576886 1602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:39.597806 1602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:39.601223 1602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:39.601437 1602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:39.602141 1602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:39.603076 1602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:39.603149 1602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.603258 1602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:39.603307 1602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.615893 1602 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:39403
I20260806 15:56:39.615919 1822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:39403 every 8 connection(s)
I20260806 15:56:39.616952 1602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:56:39.624619 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 1602
I20260806 15:56:39.624747 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:56:39.625051 14204 external_mini_cluster.cc:1428] Setting key 28195dab69176085674b71453350af7e
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:39.632194 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:39.618884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:35159 (local address 127.13.223.62:34489)
0806 15:56:39.619015 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:39.619020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:39.619996 (+ 976us) server_negotiation.cc:407] Connection header received
0806 15:56:39.621226 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:39.621229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:39.621290 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:39.621391 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:39.622498 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.623047 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.624321 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.624680 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.624786 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:39.628347 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:39.628350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:39.628352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:39.628376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:39.630106 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:39.630629 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:39.630632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:39.630634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:39.630720 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:39.631068 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:39.631073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:39.631075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:39.631257 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:39.631373 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:39.631890 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 15:56:39.632011 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:56:39.633266 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:39.619331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34489 (local address 127.13.223.3:35159)
0806 15:56:39.619830 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:39.619883 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:39.621010 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:39.621580 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:39.621587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:39.621854 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:39.622348 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:39.622362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.623176 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.623180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:39.624197 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:39.624207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:39.625556 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:39.625561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:39.625753 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:39.626667 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:39.626685 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:39.628173 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:39.630234 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:39.630240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:39.630260 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:39.630509 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:39.630828 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:39.630831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:39.630833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:39.630968 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:39.631382 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:39.631391 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:39.631655 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 15:56:39.631849 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 15:56:39.632164 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260806 15:56:39.634528 1823 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34489
I20260806 15:56:39.634780 1823 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:39.635272 1823 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:39.636369 1022 ts_manager.cc:194] Registered new tserver with Master: 9789f0ca2fd74e428a5b4e3193a1e0a0 (127.13.223.3:39403)
I20260806 15:56:39.637037 1022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:35159
I20260806 15:56:39.640542 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:56:39.641255 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 1148
I20260806 15:56:39.689484 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 1370
I20260806 15:56:39.704563 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 1602
I20260806 15:56:39.718377 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 975
I20260806 15:56:39.733141 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34489
--webserver_interface=127.13.223.62
--webserver_port=38759
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34489
--ranger_config_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:39.847613 1827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:39.847882 1827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:39.847923 1827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:39.852023 1827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:39.852088 1827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:39.852108 1827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:39.852124 1827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:39.852139 1827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:39.857497 1827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34489
--ranger_config_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:39.858472 1827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:39.859467 1827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:39.868124 1839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:39.868129 1845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:39.868103 1838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:39.868355 1827 server_base.cc:1034] running on GCE node
I20260806 15:56:39.868988 1827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:39.870154 1827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:39.871317 1827 hybrid_clock.cc:648] HybridClock initialized: now 1786031799871302 us; error 26 us; skew 500 ppm
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:39.874191 1827 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:39.875033 1827 webserver.cc:460] Webserver started at http://127.13.223.62:38759/ using document root <none> and password file <none>
I20260806 15:56:39.875545 1827 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:39.875636 1827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 15:56:39 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:39.924419 1827 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:56:39.927351 1854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:39.929075 1827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:56:39.929204 1827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "47504d804b664f2e9afa765ab51c5fee"
format_stamp: "Formatted at 2026-08-06 15:56:37 on dist-test-slave-rwrg"
server_key: "0ccd034fe17fde819a55dfb171609cfe"
server_key_iv: "f81a44e2bdfb6004754075fb2723beb7"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:39.929766 1827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:39.955279 1827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:39.958473 1827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:39.958652 1827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:39.966524 1827 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34489
I20260806 15:56:39.966528 1942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34489 every 8 connection(s)
I20260806 15:56:39.967742 1827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:39.971022 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 1827
I20260806 15:56:39.971457 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:46297
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:34489
--webserver_port=35175
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:39.972360 1943 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:56:39.974054 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Bootstrap starting.
I20260806 15:56:39.990880 1943 log.cc:824] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:39.997615 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:56:39.998054 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Bootstrap complete.
I20260806 15:56:40.002548 1943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:40.003000 1943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47504d804b664f2e9afa765ab51c5fee, State: Initialized, Role: FOLLOWER
I20260806 15:56:40.003748 1943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:40.003921 1943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:56:40.004019 1943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:56:40.004148 1943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 1 FOLLOWER]: Advancing to term 2
I20260806 15:56:40.006249 1943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:40.006616 1943 leader_election.cc:302] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47504d804b664f2e9afa765ab51c5fee; no voters:
I20260806 15:56:40.006981 1943 leader_election.cc:288] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 15:56:40.007079 1956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 15:56:40.007277 1956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [term 2 LEADER]: Becoming Leader. State: Replica: 47504d804b664f2e9afa765ab51c5fee, State: Running, Role: LEADER
I20260806 15:56:40.007579 1956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } }
I20260806 15:56:40.008293 1943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:40.009083 1957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47504d804b664f2e9afa765ab51c5fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } } }
I20260806 15:56:40.009228 1957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:40.009800 1959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47504d804b664f2e9afa765ab51c5fee. Latest consensus state: current_term: 2 leader_uuid: "47504d804b664f2e9afa765ab51c5fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47504d804b664f2e9afa765ab51c5fee" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34489 } } }
I20260806 15:56:40.009959 1959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:40.010346 1966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:40.010748 1943 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 15:56:40.017417 1966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:40.018383 1966 catalog_manager.cc:1259] Loaded cluster ID: 53b11ec0dbc24cb89426d56496639772
I20260806 15:56:40.018455 1966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:56:40.025166 1966 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:56:40.026196 1966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Loaded TSK: 0
I20260806 15:56:40.027732 1966 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 15:56:40.108893 1949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:40.109396 1949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:40.109506 1949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:40.114760 1949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:40.114863 1949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:40.114974 1949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:56:40.120676 1949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:46297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=35175
--tserver_master_addrs=127.13.223.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:40.122187 1949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:40.123672 1949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:40.135280 2019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:40.135664 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:40.135942 1949 server_base.cc:1034] running on GCE node
W20260806 15:56:40.135879 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:40.136459 1949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:40.137194 1949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:40.138428 1949 hybrid_clock.cc:648] HybridClock initialized: now 1786031800138413 us; error 65 us; skew 500 ppm
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:40.142751 1949 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:56:40.143920 1949 webserver.cc:460] Webserver started at http://127.13.223.1:35175/ using document root <none> and password file <none>
I20260806 15:56:40.144678 1949 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:40.144795 1949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:40.171082 1949 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 15:56:40.174574 2036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.176298 1949 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 15:56:40.176463 1949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "366a56dd746f4e8a9497879327eeb294"
format_stamp: "Formatted at 2026-08-06 15:56:38 on dist-test-slave-rwrg"
server_key: "a2ad4bf2f0f4e1e7ba79f488f4455057"
server_key_iv: "508dfa102c26d9dd86ed684a0f011055"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:40.177233 1949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:40.211818 1949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:40.215538 1949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:40.215763 1949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:40.216449 1949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:40.217463 1949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:40.217530 1949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.217579 1949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:40.217604 1949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.235514 1949 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:46297
I20260806 15:56:40.236702 1949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:56:40.237907 2217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:46297 every 8 connection(s)
I20260806 15:56:40.240339 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 1949
I20260806 15:56:40.240867 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:43969
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:34489
--webserver_port=34767
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:40.270357 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.240944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34489 (local address 127.13.223.1:38217)
0806 15:56:40.248817 (+ 7873us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:40.248906 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:40.250399 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:40.254857 (+ 4458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:40.254867 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:40.255276 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:40.255959 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:40.255981 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.258077 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.258081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:40.259453 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:40.259464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.260564 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.260570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:40.260779 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.261595 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:40.261618 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:40.263418 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:40.265962 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:40.265970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:40.265985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:40.266333 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:40.267479 (+ 1146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:40.267482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:40.267484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:40.267688 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:40.268523 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:40.268531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:40.268770 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 15:56:40.268969 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 15:56:40.269326 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7508,"thread_start_us":144,"threads_started":1}
I20260806 15:56:40.270344 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.240801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:38217 (local address 127.13.223.62:34489)
0806 15:56:40.252842 (+ 12041us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:40.252876 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:40.252934 (+ 58us) server_negotiation.cc:407] Connection header received
0806 15:56:40.253299 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:40.253317 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:40.253816 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:40.254258 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:40.256201 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.257888 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.259615 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.260336 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.261234 (+ 898us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.263692 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:40.263702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:40.263723 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:40.263758 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:40.265828 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.267154 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.267161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.267167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.267323 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.267816 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.267819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.267821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.268365 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:40.268526 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:40.269095 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 15:56:40.269362 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11816,"thread_start_us":129,"threads_started":1}
I20260806 15:56:40.273984 2220 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34489
I20260806 15:56:40.274238 2220 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.274816 2220 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
W20260806 15:56:40.277386 2220 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:34489 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 15:56:40.277511 2220 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.277694 2220 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:40.278622 2220 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.278795 2220 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
W20260806 15:56:40.279654 2220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 15:56:40.405474 2226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:40.405768 2226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:40.405812 2226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:40.410601 2226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:40.410686 2226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:40.410784 2226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:56:40.416989 2226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:43969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=34767
--tserver_master_addrs=127.13.223.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:40.418150 2226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:40.419232 2226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:40.432420 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:40.432714 2226 server_base.cc:1034] running on GCE node
W20260806 15:56:40.432415 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:40.432976 2242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:40.433701 2226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:40.434695 2226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:40.435946 2226 hybrid_clock.cc:648] HybridClock initialized: now 1786031800435900 us; error 71 us; skew 500 ppm
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:40.440260 2226 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:56:40.441562 2226 webserver.cc:460] Webserver started at http://127.13.223.2:34767/ using document root <none> and password file <none>
I20260806 15:56:40.442334 2226 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:40.442452 2226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:40.467576 2226 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260806 15:56:40.470876 2257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.472458 2226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:56:40.472635 2226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "a2be3674eeb54f489147f8114006ee37"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "8acc844c70c9c7f265a9f6aeea8b1a39"
server_key_iv: "9436a0122a0817e68eef80e160ac8416"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:40.473357 2226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:40.487811 2226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:40.491467 2226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:40.491707 2226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:40.492441 2226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:40.493628 2226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:40.493696 2226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.493777 2226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:40.493813 2226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.507583 2226 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:43969
I20260806 15:56:40.508984 2226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:56:40.515285 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 2226
I20260806 15:56:40.515756 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.20:46533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:39403
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:34489
--webserver_port=43917
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:40.520012 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:43969 every 8 connection(s)
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:40.528105 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.511511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36671 (local address 127.13.223.62:34489)
0806 15:56:40.511676 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:40.511681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:40.512704 (+ 1023us) server_negotiation.cc:407] Connection header received
0806 15:56:40.513714 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:40.513717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:40.513784 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:40.513891 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:40.515023 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.515829 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.518547 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.518951 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.519046 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.522824 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:40.522830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:40.522836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:40.522877 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:40.524927 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.525739 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.525745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.525750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.525851 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.526325 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.526331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.526335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.526770 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:40.526874 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:40.527690 (+ 816us) server_negotiation.cc:299] Negotiation successful
0806 15:56:40.527859 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 15:56:40.530159 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.512026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34489 (local address 127.13.223.2:36671)
0806 15:56:40.512530 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:40.512588 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:40.513499 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:40.514079 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:40.514092 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:40.514374 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:40.514847 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:40.514860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.516871 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.516878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:40.518383 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:40.518397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.519283 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.519288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:40.519517 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.520660 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:40.520686 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:40.522567 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:40.525083 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:40.525090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:40.525110 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:40.525586 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:40.525983 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:40.525989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:40.525991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:40.526194 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:40.526955 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:40.526965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:40.527341 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 15:56:40.528509 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0806 15:56:40.528911 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260806 15:56:40.531811 2449 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34489
I20260806 15:56:40.532078 2449 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.532712 2449 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
W20260806 15:56:40.534426 2449 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:34489 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 15:56:40.534555 2449 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.534729 2449 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:40.535490 2449 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.535660 2449 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
W20260806 15:56:40.536417 2449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 15:56:40.661576 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:40.661895 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:40.661969 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:40.666462 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:40.666553 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:40.666644 2454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:56:40.672663 2454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.20:46533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:39403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=43917
--tserver_master_addrs=127.13.223.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:40.673930 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:40.675076 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:40.687704 2467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:40.688184 2466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:40.688899 2469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:40.689500 2454 server_base.cc:1034] running on GCE node
I20260806 15:56:40.689988 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:40.690822 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:40.698246 2454 hybrid_clock.cc:648] HybridClock initialized: now 1786031800698198 us; error 54 us; skew 500 ppm
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:40.701354 2454 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:56:40.702441 2454 webserver.cc:460] Webserver started at http://127.13.223.3:43917/ using document root <none> and password file <none>
I20260806 15:56:40.703037 2454 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:40.703158 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:40.731724 2454 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.001s
I20260806 15:56:40.734694 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.736214 2454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:56:40.736356 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0"
format_stamp: "Formatted at 2026-08-06 15:56:39 on dist-test-slave-rwrg"
server_key: "02337781433d4aaf4d615b6f197a8554"
server_key_iv: "6143c87adb146db36064b9eb7ba91484"
server_key_version: "kuduclusterkey@0"
I20260806 15:56:40.737140 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:40.774451 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:40.778345 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:40.778568 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:40.779435 2454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:40.780534 2454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:40.780601 2454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.780669 2454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:40.780700 2454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:40.793824 2454 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:39403
I20260806 15:56:40.793850 2668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:39403 every 8 connection(s)
I20260806 15:56:40.795019 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:56:40.798120 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 2454
Aug 06 15:56:40 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:40.808354 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.798829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.13.223.62:34489)
0806 15:56:40.799223 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:40.799229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:40.799244 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:56:40.799409 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:40.799412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:40.799468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:40.799570 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:40.800364 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.801046 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.801909 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.802284 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.802394 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.803987 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:40.803991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:40.803994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:40.804019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:40.805860 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.806383 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.806387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.806389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.806490 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.806895 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.806899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.806901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.807121 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:40.807334 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:40.808032 (+ 698us) server_negotiation.cc:299] Negotiation successful
0806 15:56:40.808168 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":310,"thread_start_us":102,"threads_started":1}
I20260806 15:56:40.811633 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.797400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:60631 (local address 127.13.223.62:34489)
0806 15:56:40.797578 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:40.797583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:40.799296 (+ 1713us) server_negotiation.cc:407] Connection header received
0806 15:56:40.800545 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:40.800548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:40.800593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:40.800650 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:40.802084 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.802736 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.804402 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.804728 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.804850 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.807677 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:40.807681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:40.807683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:40.807707 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:40.809483 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.810020 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.810023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.810025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.810155 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:40.810531 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:40.810544 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:40.810545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:40.810709 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:40.810811 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:40.811354 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 15:56:40.811465 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":72}
I20260806 15:56:40.812558 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:40.797885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:34489 (local address 127.13.223.3:60631)
0806 15:56:40.799125 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:40.799184 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:40.800295 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:40.800864 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:40.800874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:40.801241 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:40.801938 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:40.801960 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.802868 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.802871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:40.804269 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:40.804281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:40.804893 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:40.804898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:40.805268 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:40.805815 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:40.805832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:40.807492 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:40.809619 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:40.809625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:40.809641 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:40.809908 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:40.810243 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:40.810248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:40.810250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:40.810430 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:40.810828 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:40.810836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:40.811122 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 15:56:40.811316 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 15:56:40.811622 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":122,"threads_started":1}
I20260806 15:56:40.813742 2673 heartbeater.cc:347] Connected to a master server at 127.13.223.62:34489
I20260806 15:56:40.814014 2673 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.814620 2673 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
W20260806 15:56:40.816370 2673 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:34489 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 15:56:40.816524 2673 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.816705 2673 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:40.817687 2673 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:40.817868 2673 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
W20260806 15:56:40.818694 2673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 15:56:41 dist-test-slave-rwrg krb5kdc[32714](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786031801, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:41.279906 2220 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:41.280225 2220 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:41.281538 1881 ts_manager.cc:194] Registered new tserver with Master: 366a56dd746f4e8a9497879327eeb294 (127.13.223.1:46297)
I20260806 15:56:41.283434 1881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:38217
I20260806 15:56:41.536614 2449 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:41.536926 2449 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:41.537767 1881 ts_manager.cc:194] Registered new tserver with Master: a2be3674eeb54f489147f8114006ee37 (127.13.223.2:43969)
I20260806 15:56:41.538558 1881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:36671
Aug 06 15:56:41 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031801, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:41.818921 2673 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:41.819252 2673 heartbeater.cc:511] Master 127.13.223.62:34489 requested a full tablet report, sending...
I20260806 15:56:41.820012 1881 ts_manager.cc:194] Registered new tserver with Master: 9789f0ca2fd74e428a5b4e3193a1e0a0 (127.13.223.3:39403)
I20260806 15:56:41.820842 1881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:60631
I20260806 15:56:41.831048 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:56:41.844713 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.834263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.13.223.62:34489)
0806 15:56:41.834898 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:41.834902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:41.834912 (+ 10us) server_negotiation.cc:407] Connection header received
0806 15:56:41.834945 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:41.834971 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:41.835009 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:41.835112 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:41.835995 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.836864 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.838075 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.838492 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.838609 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.840455 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:41.840459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:41.840462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:41.840527 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:41.842459 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:41.843009 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:41.843012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:41.843014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:41.843115 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:41.843476 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:41.843480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:41.843482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:41.843703 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:41.843822 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:41.844188 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 15:56:41.844530 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":99,"threads_started":1}
I20260806 15:56:41.852898 1881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:56:41.857053 1881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 06 15:56:41 dist-test-slave-rwrg krb5kdc[32714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031801, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for HTTP/127.13.223.20@KRBTEST.COM
I20260806 15:56:41.902029 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.893055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:43969 (local address 127.0.0.1:35790)
0806 15:56:41.895972 (+ 2917us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:41.896062 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:41.896261 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:41.897056 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:41.897061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:41.897086 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:41.897211 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:41.897218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.898724 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.898728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:41.900187 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:41.900203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.901521 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.901525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:41.901634 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.901670 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 15:56:41.901772 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 15:56:41.901881 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"thread_start_us":138,"threads_started":1}
I20260806 15:56:41.902666 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.893941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.13.223.2:43969)
0806 15:56:41.894265 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:41.894273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:41.896150 (+ 1877us) server_negotiation.cc:407] Connection header received
0806 15:56:41.896423 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:41.896429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:41.896682 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:41.896815 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:41.897340 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.898581 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.900341 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.901200 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.901301 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.901456 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:41.902384 (+ 928us) server_negotiation.cc:299] Negotiation successful
0806 15:56:41.902536 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260806 15:56:41.906486 2333 tablet_service.cc:1467] Processing CreateTablet for tablet fe52f655a2a24af38d2d70b72d2825fd (DEFAULT_TABLE table=test-table [id=31c2546516844fd7bf0eadc283feedb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:41.907996 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.895580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:39403 (local address 127.0.0.1:47888)
0806 15:56:41.903121 (+ 7541us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:41.903134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:41.903205 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:41.903868 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:41.903871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:41.903904 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:41.904020 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:41.904026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.905392 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.905395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:41.906455 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:41.906465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.907616 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.907619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:41.907718 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.907741 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:56:41.907809 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 15:56:41.907869 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7490,"thread_start_us":103,"threads_started":1}
I20260806 15:56:41.908420 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.896449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.13.223.3:39403)
0806 15:56:41.897226 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:41.897232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:41.903321 (+ 6089us) server_negotiation.cc:407] Connection header received
0806 15:56:41.903412 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:41.903417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:41.903599 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:41.903732 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:41.904145 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.905267 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.906590 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.907328 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.907424 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.907556 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:41.908193 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 15:56:41.908289 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":115,"threads_started":1}
I20260806 15:56:41.911859 2541 tablet_service.cc:1467] Processing CreateTablet for tablet fe52f655a2a24af38d2d70b72d2825fd (DEFAULT_TABLE table=test-table [id=31c2546516844fd7bf0eadc283feedb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:41.912829 2333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe52f655a2a24af38d2d70b72d2825fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:41.912997 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe52f655a2a24af38d2d70b72d2825fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:41.923938 2731 tablet_bootstrap.cc:492] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Bootstrap starting.
I20260806 15:56:41.925390 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.895298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:46297 (local address 127.0.0.1:46890)
0806 15:56:41.900656 (+ 5358us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:41.900669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:41.900790 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:41.901432 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:41.901435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:41.901452 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:41.901544 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:41.901550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.902999 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.903003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:41.923209 (+ 20206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:41.923228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.924911 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.924918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:41.925046 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.925073 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:56:41.925155 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 15:56:41.925242 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5300,"thread_start_us":116,"threads_started":1}
I20260806 15:56:41.925978 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:41.896071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.13.223.1:46297)
0806 15:56:41.897055 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:41.897062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:41.900774 (+ 3712us) server_negotiation.cc:407] Connection header received
0806 15:56:41.900926 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:41.900931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:41.901136 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:41.901291 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:41.901666 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.902870 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.923414 (+ 20544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:41.924408 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:41.924560 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:41.924831 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:41.925708 (+ 877us) server_negotiation.cc:299] Negotiation successful
0806 15:56:41.925828 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":130,"threads_started":1}
I20260806 15:56:41.928432 2731 tablet_bootstrap.cc:654] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:41.928895 2730 tablet_bootstrap.cc:492] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Bootstrap starting.
I20260806 15:56:41.929773 2731 log.cc:824] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:41.929601 2090 tablet_service.cc:1467] Processing CreateTablet for tablet fe52f655a2a24af38d2d70b72d2825fd (DEFAULT_TABLE table=test-table [id=31c2546516844fd7bf0eadc283feedb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:41.930859 2090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe52f655a2a24af38d2d70b72d2825fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:41.932732 2731 tablet_bootstrap.cc:492] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: No bootstrap required, opened a new log
I20260806 15:56:41.932938 2731 ts_tablet_manager.cc:1397] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260806 15:56:41.937788 2731 raft_consensus.cc:348] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:41.938083 2731 raft_consensus.cc:374] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:41.938154 2731 raft_consensus.cc:729] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9789f0ca2fd74e428a5b4e3193a1e0a0, State: Initialized, Role: FOLLOWER
I20260806 15:56:41.938686 2731 consensus_queue.cc:260] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:41.941032 2731 ts_tablet_manager.cc:1428] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260806 15:56:41.941453 2673 heartbeater.cc:503] Master 127.13.223.62:34489 was elected leader, sending a full tablet report...
I20260806 15:56:41.946940 2734 tablet_bootstrap.cc:492] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: Bootstrap starting.
I20260806 15:56:41.949549 2730 tablet_bootstrap.cc:654] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:41.949560 2734 tablet_bootstrap.cc:654] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:41.950630 2734 log.cc:824] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:41.951915 2730 log.cc:824] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:41.955516 2730 tablet_bootstrap.cc:492] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: No bootstrap required, opened a new log
I20260806 15:56:41.955760 2730 ts_tablet_manager.cc:1397] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20260806 15:56:41.959754 2734 tablet_bootstrap.cc:492] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: No bootstrap required, opened a new log
I20260806 15:56:41.959985 2734 ts_tablet_manager.cc:1397] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260806 15:56:41.960464 2730 raft_consensus.cc:348] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:41.960989 2730 raft_consensus.cc:374] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:41.961066 2730 raft_consensus.cc:729] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2be3674eeb54f489147f8114006ee37, State: Initialized, Role: FOLLOWER
I20260806 15:56:41.961621 2730 consensus_queue.cc:260] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:41.962833 2730 ts_tablet_manager.cc:1428] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 15:56:41.963264 2449 heartbeater.cc:503] Master 127.13.223.62:34489 was elected leader, sending a full tablet report...
I20260806 15:56:41.976312 2734 raft_consensus.cc:348] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:41.976709 2734 raft_consensus.cc:374] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:41.976780 2734 raft_consensus.cc:729] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366a56dd746f4e8a9497879327eeb294, State: Initialized, Role: FOLLOWER
I20260806 15:56:41.977335 2734 consensus_queue.cc:260] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:41.979681 2734 ts_tablet_manager.cc:1428] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20260806 15:56:41.980108 2220 heartbeater.cc:503] Master 127.13.223.62:34489 was elected leader, sending a full tablet report...
W20260806 15:56:41.991600 2222 tablet.cc:2369] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:56:42.023144 2452 tablet.cc:2369] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:42.035182 2739 raft_consensus.cc:482] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:42.035463 2739 raft_consensus.cc:504] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:42.036900 2739 leader_election.cc:288] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366a56dd746f4e8a9497879327eeb294 (127.13.223.1:46297), 9789f0ca2fd74e428a5b4e3193a1e0a0 (127.13.223.3:39403)
I20260806 15:56:42.044342 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:42.037417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36989 (local address 127.13.223.1:46297)
0806 15:56:42.037553 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:42.037557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:42.038955 (+ 1398us) server_negotiation.cc:407] Connection header received
0806 15:56:42.039164 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:42.039167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:42.039227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:42.039274 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:42.039969 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.040803 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.042855 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.043469 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.043569 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:42.043624 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:42.044106 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 15:56:42.044206 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:56:42.045163 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:42.037409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:46297 (local address 127.13.223.2:36989)
0806 15:56:42.038834 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:42.038849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:42.039042 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:42.039417 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:42.039419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:42.039436 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:42.039830 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:42.039838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.040955 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.040958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:42.042629 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:42.042641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.043757 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.043760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:42.043937 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:42.043964 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:56:42.044602 (+ 638us) client_negotiation.cc:240] Negotiation successful
0806 15:56:42.044689 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"thread_start_us":194,"threads_started":1}
I20260806 15:56:42.045943 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe52f655a2a24af38d2d70b72d2825fd" candidate_uuid: "a2be3674eeb54f489147f8114006ee37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366a56dd746f4e8a9497879327eeb294" is_pre_election: true
I20260806 15:56:42.046361 2145 raft_consensus.cc:2454] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2be3674eeb54f489147f8114006ee37 in term 0.
I20260806 15:56:42.047045 2258 leader_election.cc:302] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366a56dd746f4e8a9497879327eeb294, a2be3674eeb54f489147f8114006ee37; no voters:
I20260806 15:56:42.047436 2739 raft_consensus.cc:2781] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:56:42.047533 2739 raft_consensus.cc:482] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:56:42.047578 2739 raft_consensus.cc:3037] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:42.048919 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:42.038492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:40941 (local address 127.13.223.3:39403)
0806 15:56:42.038611 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:42.038616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:42.041899 (+ 3283us) server_negotiation.cc:407] Connection header received
0806 15:56:42.044082 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:42.044086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:42.044144 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:42.044199 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:42.044880 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.045612 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.047500 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.048097 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.048198 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:42.048248 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:42.048713 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 15:56:42.048797 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:56:42.049440 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:42.037923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:39403 (local address 127.13.223.2:40941)
0806 15:56:42.041780 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:42.041795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:42.043917 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:42.044357 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:42.044360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:42.044380 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:42.044726 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:42.044736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.045761 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.045765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:42.047345 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:42.047357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.048351 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.048354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:42.048550 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:42.048577 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:56:42.049138 (+ 561us) client_negotiation.cc:240] Negotiation successful
0806 15:56:42.049218 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3786,"thread_start_us":160,"threads_started":1}
I20260806 15:56:42.049642 2739 raft_consensus.cc:504] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:42.050048 2739 leader_election.cc:288] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [CANDIDATE]: Term 1 election: Requested vote from peers 366a56dd746f4e8a9497879327eeb294 (127.13.223.1:46297), 9789f0ca2fd74e428a5b4e3193a1e0a0 (127.13.223.3:39403)
W20260806 15:56:42.050347 2675 tablet.cc:2369] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:42.051301 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe52f655a2a24af38d2d70b72d2825fd" candidate_uuid: "a2be3674eeb54f489147f8114006ee37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" is_pre_election: true
I20260806 15:56:42.051599 2567 raft_consensus.cc:2454] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2be3674eeb54f489147f8114006ee37 in term 0.
I20260806 15:56:42.056469 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe52f655a2a24af38d2d70b72d2825fd" candidate_uuid: "a2be3674eeb54f489147f8114006ee37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366a56dd746f4e8a9497879327eeb294"
I20260806 15:56:42.056671 2145 raft_consensus.cc:3037] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:42.058717 2145 raft_consensus.cc:2454] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2be3674eeb54f489147f8114006ee37 in term 1.
I20260806 15:56:42.059087 2258 leader_election.cc:302] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366a56dd746f4e8a9497879327eeb294, a2be3674eeb54f489147f8114006ee37; no voters:
I20260806 15:56:42.061969 2739 raft_consensus.cc:2781] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:42.062275 2739 raft_consensus.cc:686] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 1 LEADER]: Becoming Leader. State: Replica: a2be3674eeb54f489147f8114006ee37, State: Running, Role: LEADER
I20260806 15:56:42.062695 2739 consensus_queue.cc:237] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } }
I20260806 15:56:42.063722 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe52f655a2a24af38d2d70b72d2825fd" candidate_uuid: "a2be3674eeb54f489147f8114006ee37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0"
I20260806 15:56:42.063877 2567 raft_consensus.cc:3037] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:42.065719 2567 raft_consensus.cc:2454] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2be3674eeb54f489147f8114006ee37 in term 1.
I20260806 15:56:42.069476 1881 catalog_manager.cc:5495] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2be3674eeb54f489147f8114006ee37 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "a2be3674eeb54f489147f8114006ee37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2be3674eeb54f489147f8114006ee37" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 43969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260806 15:56:42.122985 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:42.117399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.13.223.2:43969)
0806 15:56:42.117562 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:42.117567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:42.117584 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:56:42.117683 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:42.117686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:42.117765 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:42.117908 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:56:42.118349 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.119186 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.120622 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:42.120950 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:42.121093 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:42.121543 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:56:42.122219 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:56:42.122355 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:42.122746 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 15:56:42.122816 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:56:42.133100 2145 raft_consensus.cc:1261] T fe52f655a2a24af38d2d70b72d2825fd P 366a56dd746f4e8a9497879327eeb294 [term 1 FOLLOWER]: Refusing update from remote peer a2be3674eeb54f489147f8114006ee37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:42.134148 2752 consensus_queue.cc:1035] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "366a56dd746f4e8a9497879327eeb294" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:42.134323 2567 raft_consensus.cc:1261] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 1 FOLLOWER]: Refusing update from remote peer a2be3674eeb54f489147f8114006ee37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:42.135011 2752 consensus_queue.cc:1035] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9789f0ca2fd74e428a5b4e3193a1e0a0" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:42.147717 2761 mvcc.cc:205] Tried to move back new op lower bound from 7315586261527101440 to 7315586261263474688. Current Snapshot: MvccSnapshot[applied={T|T < 7315586261527101440 or (T in {7315586261527101440})}]
I20260806 15:56:42.150682 2757 mvcc.cc:205] Tried to move back new op lower bound from 7315586261527101440 to 7315586261263474688. Current Snapshot: MvccSnapshot[applied={T|T < 7315586261527101440}]
I20260806 15:56:42.154501 2758 mvcc.cc:205] Tried to move back new op lower bound from 7315586261527101440 to 7315586261263474688. Current Snapshot: MvccSnapshot[applied={T|T < 7315586261527101440}]
I20260806 15:56:42.178390 1881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:56:42.178664 1881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:56:42.182101 1881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47504d804b664f2e9afa765ab51c5fee: Sending DeleteTablet for 3 replicas of tablet fe52f655a2a24af38d2d70b72d2825fd
I20260806 15:56:42.183169 2090 tablet_service.cc:1514] Processing DeleteTablet for tablet fe52f655a2a24af38d2d70b72d2825fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:42 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:46890
I20260806 15:56:42.183531 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet fe52f655a2a24af38d2d70b72d2825fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:42 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:47888
I20260806 15:56:42.183920 2333 tablet_service.cc:1514] Processing DeleteTablet for tablet fe52f655a2a24af38d2d70b72d2825fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:42 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:35790
I20260806 15:56:42.184176 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 1949
I20260806 15:56:42.196893 2793 tablet_replica.cc:330] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: stopping tablet replica
I20260806 15:56:42.197304 2793 raft_consensus.cc:2229] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:56:42.197728 2793 raft_consensus.cc:2258] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:42.198238 2789 tablet_replica.cc:330] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: stopping tablet replica
I20260806 15:56:42.198572 2789 raft_consensus.cc:2229] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:56:42.198860 2789 raft_consensus.cc:2258] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:42.204247 2793 ts_tablet_manager.cc:1905] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:42.207545 2793 ts_tablet_manager.cc:1918] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:42.207644 2793 log.cc:1196] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/fe52f655a2a24af38d2d70b72d2825fd
I20260806 15:56:42.207980 2793 ts_tablet_manager.cc:1939] T fe52f655a2a24af38d2d70b72d2825fd P a2be3674eeb54f489147f8114006ee37: Deleting consensus metadata
I20260806 15:56:42.209343 1859 catalog_manager.cc:4841] TS a2be3674eeb54f489147f8114006ee37 (127.13.223.2:43969): tablet fe52f655a2a24af38d2d70b72d2825fd (table test-table [id=31c2546516844fd7bf0eadc283feedb0]) successfully deleted
I20260806 15:56:42.210366 2789 ts_tablet_manager.cc:1905] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:42.213259 2789 ts_tablet_manager.cc:1918] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:42.213370 2789 log.cc:1196] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/fe52f655a2a24af38d2d70b72d2825fd
I20260806 15:56:42.213877 2789 ts_tablet_manager.cc:1939] T fe52f655a2a24af38d2d70b72d2825fd P 9789f0ca2fd74e428a5b4e3193a1e0a0: Deleting consensus metadata
I20260806 15:56:42.215272 1856 catalog_manager.cc:4841] TS 9789f0ca2fd74e428a5b4e3193a1e0a0 (127.13.223.3:39403): tablet fe52f655a2a24af38d2d70b72d2825fd (table test-table [id=31c2546516844fd7bf0eadc283feedb0]) successfully deleted
W20260806 15:56:42.216228 1855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:46297 (error 108)
I20260806 15:56:42.217420 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:42.217121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:46297 (local address 127.0.0.1:46896)
0806 15:56:42.217260 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:42.217331 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:46297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 15:56:42.217593 1855 catalog_manager.cc:4584] TS 366a56dd746f4e8a9497879327eeb294 (127.13.223.1:46297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe52f655a2a24af38d2d70b72d2825fd: Network error: Client connection negotiation failed: client connection to 127.13.223.1:46297: connect: Connection refused (error 111)
I20260806 15:56:42.217995 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 2226
I20260806 15:56:42.231294 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 2454
I20260806 15:56:42.244805 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 1827
I20260806 15:56:42.261509 14204 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 15:56:42.607398 14204 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 15:56:42.607807 14204 mini_ranger.cc:66] Stopping Ranger...
I20260806 15:56:42.953505 14204 mini_ranger.cc:68] Stopped Ranger
2026-08-06 15:56:42.953 UTC [300] LOG: received smart shutdown request
2026-08-06 15:56:42.958 UTC [300] LOG: background worker "logical replication launcher" (PID 313) exited with exit code 1
2026-08-06 15:56:42.958 UTC [306] LOG: shutting down
2026-08-06 15:56:43.131 UTC [300] LOG: database system is shut down
2026-08-06T15:56:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50518 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:56:43 dist-test-slave-rwrg krb5kdc[2814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:56:43 dist-test-slave-rwrg krb5kdc[2814](info): set up 2 sockets
Aug 06 15:56:43 dist-test-slave-rwrg krb5kdc[2814](info): commencing operation
krb5kdc: starting...
W20260806 15:56:44.591933 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:56:44 dist-test-slave-rwrg krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:56:44.719951 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44091
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44091,127.13.223.61:43023,127.13.223.60:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:44.836197 2838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:44.836476 2838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:44.836580 2838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:44.840505 2838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:44.840557 2838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:44.840576 2838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:44.840593 2838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:44.840608 2838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:44.845836 2838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44091,127.13.223.61:43023,127.13.223.60:45001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:44.846845 2838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:44.847823 2838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:44.856521 2848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:44.856634 2851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:44.856674 2847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:44.856927 2838 server_base.cc:1034] running on GCE node
I20260806 15:56:44.857424 2838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:44.858525 2838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:44.859683 2838 hybrid_clock.cc:648] HybridClock initialized: now 1786031804859656 us; error 39 us; skew 500 ppm
Aug 06 15:56:44 dist-test-slave-rwrg krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:44.862331 2838 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:44.863163 2838 webserver.cc:460] Webserver started at http://127.13.223.62:35399/ using document root <none> and password file <none>
I20260806 15:56:44.863663 2838 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:44.863718 2838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:44.863873 2838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:44.865948 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "448452df99784f93844e8aed2b9f7962"
format_stamp: "Formatted at 2026-08-06 15:56:44 on dist-test-slave-rwrg"
server_key: "0973ae659888085fec0969c1d9f25001"
server_key_iv: "07dde5b761f9204c69832799a7a7c67b"
server_key_version: "encryptionkey@0"
I20260806 15:56:44.866513 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "448452df99784f93844e8aed2b9f7962"
format_stamp: "Formatted at 2026-08-06 15:56:44 on dist-test-slave-rwrg"
server_key: "0973ae659888085fec0969c1d9f25001"
server_key_iv: "07dde5b761f9204c69832799a7a7c67b"
server_key_version: "encryptionkey@0"
I20260806 15:56:44.870085 2838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:56:44.872385 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:44.873611 2838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:56:44.873736 2838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "448452df99784f93844e8aed2b9f7962"
format_stamp: "Formatted at 2026-08-06 15:56:44 on dist-test-slave-rwrg"
server_key: "0973ae659888085fec0969c1d9f25001"
server_key_iv: "07dde5b761f9204c69832799a7a7c67b"
server_key_version: "encryptionkey@0"
I20260806 15:56:44.873842 2838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:44.911167 2838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:44.914376 2838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:44.914583 2838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:44.922011 2838 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:44091
I20260806 15:56:44.922043 2950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:44091 every 8 connection(s)
I20260806 15:56:44.923033 2838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:44.925979 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:44.927142 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 2838
I20260806 15:56:44.927277 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:56:44.927569 14204 external_mini_cluster.cc:1428] Setting key 2359844fb2a22275c62343ebf3d87a2b
I20260806 15:56:44.930451 2953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.61@KRBTEST.COM" created.
Aug 06 15:56:44 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:44.947799 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:44.933329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.13.223.62:44091)
0806 15:56:44.933914 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:44.933938 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:44.934152 (+ 214us) server_negotiation.cc:407] Connection header received
0806 15:56:44.935655 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:44.935660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:44.935837 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:44.936133 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:44.937163 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:44.938093 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:44.939145 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:44.939501 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:44.939643 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:44.942339 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:44.942347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:44.942365 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:44.942387 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:44.944159 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:44.944780 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:44.944787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:44.944788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:44.944898 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:44.945245 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:44.945249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:44.945250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:44.945661 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:44.945813 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:44.946238 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 15:56:44.946615 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":464,"spinlock_wait_cycles":22400,"thread_start_us":205,"threads_started":1}
I20260806 15:56:44.947799 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:44.933329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44091 (local address 127.0.0.1:37308)
0806 15:56:44.933959 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:44.934024 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:44.935308 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:44.936091 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:44.936099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:44.936593 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:44.936990 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:44.937012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:44.938212 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:44.938215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:44.939033 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:44.939043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:44.939718 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:44.939722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:44.939812 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:44.940400 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:44.940422 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:44.942180 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:44.944294 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:44.944303 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:44.944314 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:44.944645 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:44.945008 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:44.945011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:44.945013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:44.945141 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:44.945782 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:44.945789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:44.945953 (+ 164us) client_negotiation.cc:769] Sending connection context
0806 15:56:44.946144 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 15:56:44.946671 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":188,"thread_start_us":151,"threads_started":1}
I20260806 15:56:44.950461 2953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:56:44.951256 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:44.950995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37800)
0806 15:56:44.951120 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:44.951195 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 15:56:44.951427 2863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
I20260806 15:56:44.952239 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:44.952026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37814)
0806 15:56:44.952147 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:44.952188 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 15:56:44.952368 2953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:43023: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab.
Entry for principal kudu/127.13.223.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab.
I20260806 15:56:44.956101 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:43023
--webserver_interface=127.13.223.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab
--principal=kudu/127.13.223.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44091,127.13.223.61:43023,127.13.223.60:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:45.001689 2953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } attempt: 1
I20260806 15:56:45.002642 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.002311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37822)
0806 15:56:45.002464 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.002531 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 15:56:45.003430 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.003214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37826)
0806 15:56:45.003320 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.003371 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 15:56:45.003605 2953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:43023: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
I20260806 15:56:45.071780 2953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } attempt: 2
I20260806 15:56:45.072655 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.072331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37836)
0806 15:56:45.072477 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.072568 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 15:56:45.073521 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.073263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37842)
0806 15:56:45.073401 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.073459 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 15:56:45.073747 2953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:43023: Network error: Client connection negotiation failed: client connection to 127.13.223.61:43023: connect: Connection refused (error 111)
W20260806 15:56:45.073774 2964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:45.074056 2964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:45.074142 2964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:45.078380 2964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:45.078469 2964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:45.078528 2964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:45.078578 2964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:45.078611 2964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:45.084134 2964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44091,127.13.223.61:43023,127.13.223.60:45001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab
--principal=kudu/127.13.223.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:45.085278 2964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:45.086299 2964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:45.094751 2976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:45.094761 2974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:45.094761 2973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:45.095063 2964 server_base.cc:1034] running on GCE node
I20260806 15:56:45.095590 2964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:45.096745 2964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:45.097911 2964 hybrid_clock.cc:648] HybridClock initialized: now 1786031805097894 us; error 30 us; skew 500 ppm
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:45.100912 2964 init.cc:377] Logged in from keytab as kudu/127.13.223.61@KRBTEST.COM (short username kudu)
I20260806 15:56:45.101785 2964 webserver.cc:460] Webserver started at http://127.13.223.61:34749/ using document root <none> and password file <none>
I20260806 15:56:45.102295 2964 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:45.102353 2964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:45.102516 2964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:45.104863 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data/instance:
uuid: "0df87bca49314aa58422074c4ba5ab1c"
format_stamp: "Formatted at 2026-08-06 15:56:45 on dist-test-slave-rwrg"
server_key: "bee6fe54074c2905927e6341465bd90c"
server_key_iv: "aa1665c009287396499dc8de93af899a"
server_key_version: "encryptionkey@0"
I20260806 15:56:45.105423 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal/instance:
uuid: "0df87bca49314aa58422074c4ba5ab1c"
format_stamp: "Formatted at 2026-08-06 15:56:45 on dist-test-slave-rwrg"
server_key: "bee6fe54074c2905927e6341465bd90c"
server_key_iv: "aa1665c009287396499dc8de93af899a"
server_key_version: "encryptionkey@0"
I20260806 15:56:45.109560 2964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:56:45.112222 2985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:45.113636 2964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:56:45.113770 2964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal
uuid: "0df87bca49314aa58422074c4ba5ab1c"
format_stamp: "Formatted at 2026-08-06 15:56:45 on dist-test-slave-rwrg"
server_key: "bee6fe54074c2905927e6341465bd90c"
server_key_iv: "aa1665c009287396499dc8de93af899a"
server_key_version: "encryptionkey@0"
I20260806 15:56:45.113878 2964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:45.126405 2964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:45.129766 2964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:45.129976 2964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:45.137542 2964 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.61:43023
I20260806 15:56:45.137598 3066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.61:43023 every 8 connection(s)
I20260806 15:56:45.138494 2964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
I20260806 15:56:45.141367 3067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:45.143033 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 2964
I20260806 15:56:45.143186 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-1/wal/instance
I20260806 15:56:45.143479 14204 external_mini_cluster.cc:1428] Setting key 94ccd47e2d66032fb854496b6c71f326
I20260806 15:56:45.145380 3067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.60@KRBTEST.COM" created.
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:45.162086 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.147307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.13.223.62:44091)
0806 15:56:45.147496 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.147504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.148359 (+ 855us) server_negotiation.cc:407] Connection header received
0806 15:56:45.149387 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.149391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.149455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.149554 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.150734 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.151253 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.152449 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.152749 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.152861 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.156322 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.156325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.156328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.156354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.158252 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.159106 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.159110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.159115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.159285 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.159792 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.159798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.159800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.160203 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.160313 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.161798 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.161945 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:56:45.163455 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.147692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44091 (local address 127.0.0.1:37322)
0806 15:56:45.148200 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.148254 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.149167 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.149761 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.149768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.150106 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.150576 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.150590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.151384 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.151387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.152325 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.152335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.152905 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.152909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.153152 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.153733 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.153749 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.156027 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.158417 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.158424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.158442 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.158912 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.159410 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.159416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.159419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.159664 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.160388 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.160396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.160792 (+ 396us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.162299 (+ 1507us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.162622 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
I20260806 15:56:45.164906 2953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } attempt: 3
I20260806 15:56:45.165139 3067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab.
Entry for principal kudu/127.13.223.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab.
I20260806 15:56:45.173799 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:45001
--webserver_interface=127.13.223.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab
--principal=kudu/127.13.223.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44091,127.13.223.61:43023,127.13.223.60:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:45.176088 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.165355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37858)
0806 15:56:45.165507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.165524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.165647 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.166401 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.166404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.166647 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.166745 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.166750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.167905 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.167908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.168471 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.168479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.169396 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.169399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.169522 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.170053 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.170065 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.171627 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.174371 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.174377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.174383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.174665 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.174946 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.174951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.174956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.175061 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.175515 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.175518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.175637 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.175746 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.175894 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 15:56:45.176312 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.165658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.13.223.61:43023)
0806 15:56:45.166010 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.166025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.166048 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:56:45.166085 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.166092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.166279 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.166415 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.166859 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.167791 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.168653 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.169277 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.169380 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.171772 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.171782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.171795 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.171816 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.174261 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.174770 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.174775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.174776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.174857 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.175158 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.175160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.175161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.175403 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.175547 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.175852 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.176059 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":269,"thread_start_us":151,"threads_started":1}
I20260806 15:56:45.178426 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.165700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37868)
0806 15:56:45.165984 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.165998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.166076 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.166483 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.166486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.166786 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.166889 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.166895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.167903 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.167906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.168457 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.168465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.169802 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.169806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.169920 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.170496 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.170510 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.172123 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.176795 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.176798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.176801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.177054 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.177365 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.177368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.177369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.177499 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.177953 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.177956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.178046 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.178152 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.178275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260806 15:56:45.178628 2953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:56:45.178684 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.165717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.13.223.61:43023)
0806 15:56:45.166217 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.166224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.166237 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:56:45.166271 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.166273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.166310 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.166447 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.167023 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.167761 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.168658 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.169278 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.169391 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.172284 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.172291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.172294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.172322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.176681 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.177160 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.177166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.177171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.177265 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.177592 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.177596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.177598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.177839 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.177940 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.178228 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.178370 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":439,"thread_start_us":173,"threads_started":1}
I20260806 15:56:45.179329 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.179094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51666)
0806 15:56:45.179194 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.179254 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 15:56:45.179832 3067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:56:45.180065 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.179790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51678)
0806 15:56:45.179946 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.179998 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 15:56:45.180439 2953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:45001: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
I20260806 15:56:45.180586 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.180308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51684)
0806 15:56:45.180411 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.180481 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 15:56:45.180784 2986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
I20260806 15:56:45.181591 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.181389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51690)
0806 15:56:45.181489 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.181532 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 15:56:45.181740 3067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:45001: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
I20260806 15:56:45.211617 2953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } attempt: 1
I20260806 15:56:45.212544 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.212215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51706)
0806 15:56:45.212368 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.212432 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 15:56:45.213245 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.213035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51712)
0806 15:56:45.213142 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.213192 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 15:56:45.213370 2953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:45001: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
I20260806 15:56:45.230975 3067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } attempt: 1
I20260806 15:56:45.231835 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.231504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51720)
0806 15:56:45.231672 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.231745 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 15:56:45.232580 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.232336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51722)
0806 15:56:45.232448 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.232519 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 15:56:45.232728 3067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:45001: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
I20260806 15:56:45.288550 2953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } attempt: 2
I20260806 15:56:45.289350 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.289047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51736)
0806 15:56:45.289184 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.289257 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 15:56:45.290129 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.289914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51750)
0806 15:56:45.290018 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.290074 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 15:56:45.290287 2953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:45001: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
W20260806 15:56:45.290894 3075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:45.291112 3075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:45.291193 3075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:45.295156 3075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:45.295210 3075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:45.295231 3075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:45.295301 3075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:45.295329 3075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:45.300439 3075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44091,127.13.223.61:43023,127.13.223.60:45001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab
--principal=kudu/127.13.223.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:45001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:45.300864 3067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } attempt: 2
I20260806 15:56:45.301620 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.301316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51754)
0806 15:56:45.301468 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.301538 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 15:56:45.301714 3075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:45.302445 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.302211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51756)
0806 15:56:45.302326 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.302380 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 15:56:45.302582 3067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:45001: Network error: Client connection negotiation failed: client connection to 127.13.223.60:45001: connect: Connection refused (error 111)
I20260806 15:56:45.302866 3075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:45.311028 3089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:45.311000 3091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:45.311000 3086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:45.311412 3075 server_base.cc:1034] running on GCE node
I20260806 15:56:45.311940 3075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:45.313056 3075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:45.314253 3075 hybrid_clock.cc:648] HybridClock initialized: now 1786031805314220 us; error 41 us; skew 500 ppm
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:45.317016 3075 init.cc:377] Logged in from keytab as kudu/127.13.223.60@KRBTEST.COM (short username kudu)
I20260806 15:56:45.317826 3075 webserver.cc:460] Webserver started at http://127.13.223.60:40407/ using document root <none> and password file <none>
I20260806 15:56:45.318370 3075 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:45.318430 3075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:45.318629 3075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:45.320750 3075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data/instance:
uuid: "98981860bf1849ffaa682db145d45ecc"
format_stamp: "Formatted at 2026-08-06 15:56:45 on dist-test-slave-rwrg"
server_key: "5d9e56d84c8fe7e56f27381e3f09f612"
server_key_iv: "c46bbe6775532b8c57cc7fe1cea5bb38"
server_key_version: "encryptionkey@0"
I20260806 15:56:45.321326 3075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal/instance:
uuid: "98981860bf1849ffaa682db145d45ecc"
format_stamp: "Formatted at 2026-08-06 15:56:45 on dist-test-slave-rwrg"
server_key: "5d9e56d84c8fe7e56f27381e3f09f612"
server_key_iv: "c46bbe6775532b8c57cc7fe1cea5bb38"
server_key_version: "encryptionkey@0"
I20260806 15:56:45.325439 3075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:56:45.327709 3106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:45.328969 3075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:56:45.329111 3075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal
uuid: "98981860bf1849ffaa682db145d45ecc"
format_stamp: "Formatted at 2026-08-06 15:56:45 on dist-test-slave-rwrg"
server_key: "5d9e56d84c8fe7e56f27381e3f09f612"
server_key_iv: "c46bbe6775532b8c57cc7fe1cea5bb38"
server_key_version: "encryptionkey@0"
I20260806 15:56:45.329222 3075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:45.348650 3075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:45.352138 3075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:45.352347 3075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:45.359629 3075 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.60:45001
I20260806 15:56:45.359660 3203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.60:45001 every 8 connection(s)
I20260806 15:56:45.360622 3075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
I20260806 15:56:45.361907 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 3075
I20260806 15:56:45.362073 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786031632208048-14204-0/minicluster-data/master-2/wal/instance
I20260806 15:56:45.362366 14204 external_mini_cluster.cc:1428] Setting key 77b47cf266a5cdcf450d12341523dc38
I20260806 15:56:45.363638 3204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:45.367520 3204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:45.377372 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.363806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.13.223.60:45001)
0806 15:56:45.364275 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.364291 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.364329 (+ 38us) server_negotiation.cc:407] Connection header received
0806 15:56:45.365096 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.365114 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.365459 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.365810 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.366670 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.367696 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.368686 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.369367 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.370020 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.372273 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.372279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.372288 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.372317 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.374343 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.374767 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.374774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.374780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.374941 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.375262 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.375265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.375266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.375740 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.375988 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.376245 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.376582 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":150,"threads_started":1}
I20260806 15:56:45.379000 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.369377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.13.223.62:44091)
0806 15:56:45.369516 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.369523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.369850 (+ 327us) server_negotiation.cc:407] Connection header received
0806 15:56:45.370066 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.370072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.370134 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.370191 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.371026 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.371509 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.372605 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.372827 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.372939 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.375502 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.375506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.375508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.375531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.377066 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.377685 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.377689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.377690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.377776 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.378045 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.378049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.378050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.378240 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.378343 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.378719 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.378853 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 15:56:45.379012 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.369309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44091 (local address 127.0.0.1:37334)
0806 15:56:45.369717 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.369756 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.369923 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.370381 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.370386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.370762 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.370891 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.370896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.371645 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.371648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.372446 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.372457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.372944 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.372947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.373076 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.373695 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.373707 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.375368 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.377177 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.377188 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.377194 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.377582 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.377865 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.377869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.377871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.377956 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.378363 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.378371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.378490 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.378647 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.378839 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":152,"threads_started":1}
I20260806 15:56:45.380412 3204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
I20260806 15:56:45.389108 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.379828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.13.223.62:44091)
0806 15:56:45.380136 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.380139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.380148 (+ 9us) server_negotiation.cc:407] Connection header received
0806 15:56:45.380175 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.380177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.380207 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.380286 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.381150 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.381669 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.382420 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.382627 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.382729 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.385437 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.385446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.385447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.385487 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.387212 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.387723 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.387725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.387726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.387799 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.388142 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.388145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.388146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.388373 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.388620 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.388833 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.388960 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260806 15:56:45.389196 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.380945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:43023 (local address 127.0.0.1:37872)
0806 15:56:45.381089 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.381118 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.381193 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.381439 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.381442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.381618 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.381707 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.381713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.382480 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.382486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.383073 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.383085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.383546 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.383549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.383625 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.384063 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.384085 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.385690 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.387499 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.387503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.387507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.387769 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.388083 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.388088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.388090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.388190 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.388685 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.388688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.388797 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.388905 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.389028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 15:56:45.389396 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.381062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.13.223.61:43023)
0806 15:56:45.381208 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.381215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.381234 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:56:45.381284 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.381287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.381336 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.381425 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.381832 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.382363 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.383197 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.383428 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.383509 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.385848 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.385851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.385852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.385895 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.387402 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.387861 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.387867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.387870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.387951 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.388297 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.388301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.388303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.388544 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.388637 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.388994 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.389149 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:56:45.389464 2953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } attempt: 3
I20260806 15:56:45.390300 3204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:56:45.393985 3067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } attempt: 3
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:56:45 dist-test-slave-rwrg krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
I20260806 15:56:45.400247 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.390033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51774)
0806 15:56:45.390158 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.390175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.390242 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.391365 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.391370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.391554 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.391648 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.391658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.392649 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.392654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.393185 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.393193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.393725 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.393728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.393815 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.394833 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.394846 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.396348 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.398425 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.398430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.398434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.398691 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.399054 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.399057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.399059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.399157 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.399643 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.399647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.399766 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.399865 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.400023 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 15:56:45.400874 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.390372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.13.223.60:45001)
0806 15:56:45.391147 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.391153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.391166 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:56:45.391201 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.391204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.391252 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.391330 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.391781 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.392537 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.393305 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.393590 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.393696 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.396557 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.396561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.396563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.396589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.398300 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.398819 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.398823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.398826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.398921 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.399283 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.399287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.399290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.399512 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.400446 (+ 934us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.400560 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.400681 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":696}
I20260806 15:56:45.402421 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.391036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.13.223.61:43023)
0806 15:56:45.391135 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.391139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.391252 (+ 113us) server_negotiation.cc:407] Connection header received
0806 15:56:45.391415 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.391418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.391453 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.391499 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.392278 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.393225 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.394299 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.394627 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.394747 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.398197 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.398201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.398203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.398286 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.400230 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.401108 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.401111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.401113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.401206 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.401528 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.401532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.401539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.401783 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.401994 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.402128 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.402240 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 15:56:45.403060 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.390778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51784)
0806 15:56:45.391064 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.391077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.391152 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.391501 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.391504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.391711 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.391837 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.391843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.393351 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.393354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.394176 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.394186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.394870 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.394876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.394970 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.396432 (+ 1462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.396446 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.398955 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.401473 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.401476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.401477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.401804 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.402089 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.402092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.402093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.402180 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.402623 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.402625 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.402697 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.402782 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.402895 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":27}
I20260806 15:56:45.403550 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.390835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.13.223.60:45001)
0806 15:56:45.391219 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.391227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.391238 (+ 11us) server_negotiation.cc:407] Connection header received
0806 15:56:45.391263 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.391265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.391318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.391393 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.391935 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.393234 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.394308 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.394681 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.394773 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.399077 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.399080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.399082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.399106 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.401366 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.401909 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.401912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.401913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.401991 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.402282 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.402288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.402292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.402498 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.402590 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.403255 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.403364 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":103,"threads_started":1}
I20260806 15:56:45.404798 2953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: Bootstrap starting.
I20260806 15:56:45.406157 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:56:45.407119 3204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: Bootstrap starting.
I20260806 15:56:45.407799 2953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:45.408775 2953 log.cc:824] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:45.409770 3204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:45.410658 3204 log.cc:824] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:45.411082 2953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: No bootstrap required, opened a new log
I20260806 15:56:45.413745 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.394343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:45001 (local address 127.0.0.1:51792)
0806 15:56:45.394474 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:45.394489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:45.394572 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:45.395522 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:45.395525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:45.395701 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:45.395813 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.395820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.396960 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.396963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.397550 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:45.397565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.398206 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.398209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:45.398295 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.398721 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:45.398731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:45.400276 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:45.411998 (+ 11722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.412002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.412004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.412279 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.412631 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:45.412634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:45.412636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:45.412735 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:45.413185 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:45.413188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:45.413288 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 15:56:45.413397 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 15:56:45.413550 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 15:56:45.414436 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.394549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.13.223.60:45001)
0806 15:56:45.395206 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.395218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.395249 (+ 31us) server_negotiation.cc:407] Connection header received
0806 15:56:45.395286 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.395289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.395333 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.395475 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.395957 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.396811 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.397724 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.398096 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.398503 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.400576 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.400582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.400585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.400615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.411870 (+ 11255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.412404 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.412407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.412409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.412518 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.412854 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.412857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.412858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.413073 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.414030 (+ 957us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.414130 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.414261 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":120,"threads_started":1}
I20260806 15:56:45.415549 2953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.415841 2953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:45.415900 2953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448452df99784f93844e8aed2b9f7962, State: Initialized, Role: FOLLOWER
I20260806 15:56:45.416445 2953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.417789 3067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: Bootstrap starting.
I20260806 15:56:45.418223 2953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:45.419492 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.408551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.13.223.62:44091)
0806 15:56:45.408674 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.408678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.409434 (+ 756us) server_negotiation.cc:407] Connection header received
0806 15:56:45.409650 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.409654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.409706 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.409759 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.410741 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.411460 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.412649 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.412970 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.413047 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.415358 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.415362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.415364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.415391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.417228 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.417844 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.417848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.417850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.417938 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.418382 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.418404 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.418406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.418633 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.418737 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.419209 (+ 472us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.419338 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:56:45.419932 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.409008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.13.223.61:43023)
0806 15:56:45.409165 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.409169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.410047 (+ 878us) server_negotiation.cc:407] Connection header received
0806 15:56:45.410226 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.410230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.410273 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.410322 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.411462 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.412196 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.413360 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.414019 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.414108 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.416073 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.416076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.416079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.416103 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.418036 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.418601 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.418605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.418606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.418693 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.419033 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.419037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.419040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.419252 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.419345 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.419702 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.419805 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 15:56:45.423627 3243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:56:45.423699 3243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:56:45.424077 3204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: No bootstrap required, opened a new log
I20260806 15:56:45.423249 2838 master_runner.cc:186] Error getting master registration for 127.13.223.61:43023: OK, instance_id { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" instance_seqno: 1786031805135399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 15:56:45.426527 2838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 15:56:45.426808 3067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:45.427497 3067 log.cc:824] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:45.420703 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.427939 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:56:45.429412 3204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.429726 3204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:45.429786 3204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98981860bf1849ffaa682db145d45ecc, State: Initialized, Role: FOLLOWER
I20260806 15:56:45.430248 3067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: No bootstrap required, opened a new log
I20260806 15:56:45.430285 3204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.431744 3204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:45.431975 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:45.409553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.13.223.60:45001)
0806 15:56:45.409728 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:45.409732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:45.421098 (+ 11366us) server_negotiation.cc:407] Connection header received
0806 15:56:45.421322 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:45.421325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:45.421384 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:45.421434 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:45.423597 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.424297 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.425469 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:45.425800 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:45.425874 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:45.427889 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:45.427896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:45.427898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:45.427931 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:45.429790 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.430414 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.430420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.430424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.430532 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:45.430937 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:45.430940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:45.430941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:45.431173 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:45.431330 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:45.431711 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 15:56:45.431837 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 15:56:45.435423 3067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.435676 3067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:45.435743 3067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df87bca49314aa58422074c4ba5ab1c, State: Initialized, Role: FOLLOWER
I20260806 15:56:45.436299 3067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
W20260806 15:56:45.437270 3271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:56:45.437395 3271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:56:45.439309 3067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:45.438287 3075 master_runner.cc:186] Error getting master registration for 127.13.223.61:43023: OK, instance_id { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" instance_seqno: 1786031805135399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 15:56:45.441679 3075 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 15:56:45.442334 3276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.442479 3276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:56:45.442832 3249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.442965 3249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: This master's current role is: FOLLOWER
W20260806 15:56:45.445531 3290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:56:45.445667 3290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:56:45.449510 2964 master_runner.cc:263] No leader master found from master 0df87bca49314aa58422074c4ba5ab1c
I20260806 15:56:45.449599 2964 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 15:56:45.603955 3115 tablet.cc:2369] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:56:45.631544 2995 tablet.cc:2369] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:56:45.666585 2868 tablet.cc:2369] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:45.680511 3229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:45.680761 3229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.681236 3249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:45.681416 3249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.681932 3229 leader_election.cc:288] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0df87bca49314aa58422074c4ba5ab1c (127.13.223.61:43023), 98981860bf1849ffaa682db145d45ecc (127.13.223.60:45001)
I20260806 15:56:45.682297 3249 leader_election.cc:288] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 448452df99784f93844e8aed2b9f7962 (127.13.223.62:44091), 0df87bca49314aa58422074c4ba5ab1c (127.13.223.61:43023)
I20260806 15:56:45.682596 3155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "448452df99784f93844e8aed2b9f7962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98981860bf1849ffaa682db145d45ecc" is_pre_election: true
I20260806 15:56:45.682595 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "448452df99784f93844e8aed2b9f7962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df87bca49314aa58422074c4ba5ab1c" is_pre_election: true
I20260806 15:56:45.682858 3023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98981860bf1849ffaa682db145d45ecc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df87bca49314aa58422074c4ba5ab1c" is_pre_election: true
I20260806 15:56:45.682859 3155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448452df99784f93844e8aed2b9f7962 in term 0.
I20260806 15:56:45.682838 2903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98981860bf1849ffaa682db145d45ecc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448452df99784f93844e8aed2b9f7962" is_pre_election: true
I20260806 15:56:45.682955 3024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448452df99784f93844e8aed2b9f7962 in term 0.
I20260806 15:56:45.683034 2903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98981860bf1849ffaa682db145d45ecc in term 0.
I20260806 15:56:45.683360 2863 leader_election.cc:302] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df87bca49314aa58422074c4ba5ab1c, 448452df99784f93844e8aed2b9f7962; no voters:
I20260806 15:56:45.683442 3112 leader_election.cc:302] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448452df99784f93844e8aed2b9f7962, 98981860bf1849ffaa682db145d45ecc; no voters:
I20260806 15:56:45.683640 3229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:56:45.683719 3229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:56:45.683765 3229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:45.683738 3249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:56:45.683820 3249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:56:45.683881 3249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:45.684970 3229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.684978 3249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.685303 3249 leader_election.cc:288] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [CANDIDATE]: Term 1 election: Requested vote from peers 448452df99784f93844e8aed2b9f7962 (127.13.223.62:44091), 0df87bca49314aa58422074c4ba5ab1c (127.13.223.61:43023)
I20260806 15:56:45.685333 3229 leader_election.cc:288] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [CANDIDATE]: Term 1 election: Requested vote from peers 0df87bca49314aa58422074c4ba5ab1c (127.13.223.61:43023), 98981860bf1849ffaa682db145d45ecc (127.13.223.60:45001)
I20260806 15:56:45.685650 3023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "448452df99784f93844e8aed2b9f7962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df87bca49314aa58422074c4ba5ab1c"
I20260806 15:56:45.685662 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98981860bf1849ffaa682db145d45ecc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df87bca49314aa58422074c4ba5ab1c"
I20260806 15:56:45.685847 3023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:45.685890 3155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "448452df99784f93844e8aed2b9f7962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98981860bf1849ffaa682db145d45ecc"
I20260806 15:56:45.686040 3155 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 448452df99784f93844e8aed2b9f7962 in current term 1: Already voted for candidate 98981860bf1849ffaa682db145d45ecc in this term.
I20260806 15:56:45.686028 2903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98981860bf1849ffaa682db145d45ecc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448452df99784f93844e8aed2b9f7962"
I20260806 15:56:45.686193 2903 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 98981860bf1849ffaa682db145d45ecc in current term 1: Already voted for candidate 448452df99784f93844e8aed2b9f7962 in this term.
I20260806 15:56:45.686630 3112 leader_election.cc:302] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98981860bf1849ffaa682db145d45ecc; no voters: 0df87bca49314aa58422074c4ba5ab1c, 448452df99784f93844e8aed2b9f7962
I20260806 15:56:45.686807 3249 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 15:56:45.687242 3023 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448452df99784f93844e8aed2b9f7962 in term 1.
I20260806 15:56:45.687510 2863 leader_election.cc:302] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0df87bca49314aa58422074c4ba5ab1c, 448452df99784f93844e8aed2b9f7962; no voters: 98981860bf1849ffaa682db145d45ecc
I20260806 15:56:45.687696 3229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:45.687992 3229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [term 1 LEADER]: Becoming Leader. State: Replica: 448452df99784f93844e8aed2b9f7962, State: Running, Role: LEADER
I20260806 15:56:45.688340 3229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } }
I20260806 15:56:45.690609 3296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448452df99784f93844e8aed2b9f7962. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.690748 3296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:45.691154 3298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:45.691983 3298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:45.696593 3023 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [term 1 FOLLOWER]: Refusing update from remote peer 448452df99784f93844e8aed2b9f7962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:45.696624 3155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [term 1 FOLLOWER]: Refusing update from remote peer 448452df99784f93844e8aed2b9f7962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:45.697284 3229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:45.697456 3296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:45.700963 3276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448452df99784f93844e8aed2b9f7962. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.701105 3276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:56:45.701141 3249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448452df99784f93844e8aed2b9f7962. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.701239 3249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:56:45.702741 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.702860 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:45.703116 3301 mvcc.cc:205] Tried to move back new op lower bound from 7315586276126932992 to 7315586276102684672. Current Snapshot: MvccSnapshot[applied={T|T < 7315586276126932992}]
I20260806 15:56:45.703277 3276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.703362 3276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:56:45.703608 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.703711 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:45.704355 3298 catalog_manager.cc:1347] Generated new cluster ID: 03d2459a0bbb429f9c9566f45b90246a
I20260806 15:56:45.704417 3298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:56:45.704397 3249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "448452df99784f93844e8aed2b9f7962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448452df99784f93844e8aed2b9f7962" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44091 } } peers { permanent_uuid: "0df87bca49314aa58422074c4ba5ab1c" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 43023 } } peers { permanent_uuid: "98981860bf1849ffaa682db145d45ecc" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 45001 } } }
I20260806 15:56:45.704540 3249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:56:45.705375 3302 mvcc.cc:205] Tried to move back new op lower bound from 7315586276126932992 to 7315586276102684672. Current Snapshot: MvccSnapshot[applied={T|T < 7315586276126932992}]
I20260806 15:56:45.723793 3298 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:56:45.725056 3298 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:56:45.731649 3298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448452df99784f93844e8aed2b9f7962: Generated new TSK 0
I20260806 15:56:45.732749 3298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:56:46.438836 3271 catalog_manager.cc:1259] Loaded cluster ID: 03d2459a0bbb429f9c9566f45b90246a
I20260806 15:56:46.438953 3271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: loading cluster ID for follower catalog manager: success
I20260806 15:56:46.440925 3271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: acquiring CA information for follower catalog manager: success
I20260806 15:56:46.442193 3271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98981860bf1849ffaa682db145d45ecc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 15:56:46.446910 3290 catalog_manager.cc:1259] Loaded cluster ID: 03d2459a0bbb429f9c9566f45b90246a
I20260806 15:56:46.447003 3290 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: loading cluster ID for follower catalog manager: success
I20260806 15:56:46.448860 3290 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: acquiring CA information for follower catalog manager: success
I20260806 15:56:46.450100 3290 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0df87bca49314aa58422074c4ba5ab1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 15:56:46.793886 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 2838
I20260806 15:56:46.806537 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 2964
I20260806 15:56:46.818967 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 3075
2026-08-06T15:56:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3568 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:56:46 dist-test-slave-rwrg krb5kdc[3326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:56:46 dist-test-slave-rwrg krb5kdc[3326](info): set up 2 sockets
Aug 06 15:56:46 dist-test-slave-rwrg krb5kdc[3326](info): commencing operation
krb5kdc: starting...
W20260806 15:56:48.081431 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:56:48 dist-test-slave-rwrg krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:56:48.216004 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34027
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:48.327376 3350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:48.327679 3350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:48.327756 3350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:48.331701 3350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:48.331797 3350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:48.331836 3350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:48.331890 3350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:48.331940 3350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:48.337244 3350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:48.338320 3350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:48.339351 3350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:48.348245 3359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:48.348276 3350 server_base.cc:1034] running on GCE node
W20260806 15:56:48.348237 3361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:48.348237 3358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:48.349284 3350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:48.350418 3350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:48.351589 3350 hybrid_clock.cc:648] HybridClock initialized: now 1786031808351576 us; error 25 us; skew 500 ppm
Aug 06 15:56:48 dist-test-slave-rwrg krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031808, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:48.358441 3350 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:48.359396 3350 webserver.cc:460] Webserver started at http://127.13.223.62:44405/ using document root <none> and password file <none>
I20260806 15:56:48.359951 3350 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:48.360011 3350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:48.360255 3350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:48.362375 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "d63fcd4b64fa4052934d42805771d0a4"
format_stamp: "Formatted at 2026-08-06 15:56:48 on dist-test-slave-rwrg"
server_key: "6883822623920d6b5a52c14fe5d12fcd"
server_key_iv: "9bbed2207f60fe9529a596480ccbd8f3"
server_key_version: "encryptionkey@0"
I20260806 15:56:48.362923 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "d63fcd4b64fa4052934d42805771d0a4"
format_stamp: "Formatted at 2026-08-06 15:56:48 on dist-test-slave-rwrg"
server_key: "6883822623920d6b5a52c14fe5d12fcd"
server_key_iv: "9bbed2207f60fe9529a596480ccbd8f3"
server_key_version: "encryptionkey@0"
I20260806 15:56:48.366768 3350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:56:48.369284 3371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:48.370580 3350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:56:48.370736 3350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "d63fcd4b64fa4052934d42805771d0a4"
format_stamp: "Formatted at 2026-08-06 15:56:48 on dist-test-slave-rwrg"
server_key: "6883822623920d6b5a52c14fe5d12fcd"
server_key_iv: "9bbed2207f60fe9529a596480ccbd8f3"
server_key_version: "encryptionkey@0"
I20260806 15:56:48.370858 3350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:48.389626 3350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:48.392853 3350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:48.393055 3350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:48.400552 3350 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:34027
I20260806 15:56:48.400575 3450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:34027 every 8 connection(s)
I20260806 15:56:48.401553 3350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:48.402899 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 3350
I20260806 15:56:48.403018 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:56:48.403299 14204 external_mini_cluster.cc:1428] Setting key 42a9a80c09b827417078eb65cffb05e7
I20260806 15:56:48.404673 3453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:48.411157 3453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4: Bootstrap starting.
Aug 06 15:56:48 dist-test-slave-rwrg krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:48.413625 3453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:48.414512 3453 log.cc:824] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:48.416376 3453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4: No bootstrap required, opened a new log
I20260806 15:56:48.417550 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:48.404660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.13.223.62:34027)
0806 15:56:48.405117 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:48.405141 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:48.405183 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:56:48.405811 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:48.405827 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:48.406153 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:48.406415 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:48.407330 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:48.408324 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:48.409513 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:48.409979 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:48.410096 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:48.412953 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:48.412958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:48.412971 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:48.412998 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:48.414815 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:48.415247 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:48.415253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:48.415263 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:48.415376 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:48.415669 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:48.415673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:48.415674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:48.416044 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:48.416253 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:48.416589 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 15:56:48.416859 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":192,"threads_started":1}
I20260806 15:56:48.420434 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:56:48.420524 3453 raft_consensus.cc:348] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63fcd4b64fa4052934d42805771d0a4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34027 } }
I20260806 15:56:48.420778 3453 raft_consensus.cc:374] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:48.420833 3453 raft_consensus.cc:729] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63fcd4b64fa4052934d42805771d0a4, State: Initialized, Role: FOLLOWER
I20260806 15:56:48.421216 3453 consensus_queue.cc:260] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63fcd4b64fa4052934d42805771d0a4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34027 } }
I20260806 15:56:48.421377 3453 raft_consensus.cc:388] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:56:48.421486 3453 raft_consensus.cc:482] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:56:48.421597 3453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:48.422621 3453 raft_consensus.cc:504] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63fcd4b64fa4052934d42805771d0a4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34027 } }
I20260806 15:56:48.422945 3453 leader_election.cc:302] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d63fcd4b64fa4052934d42805771d0a4; no voters:
I20260806 15:56:48.423240 3453 leader_election.cc:288] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:56:48.423358 3459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:48.423635 3459 raft_consensus.cc:686] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [term 1 LEADER]: Becoming Leader. State: Replica: d63fcd4b64fa4052934d42805771d0a4, State: Running, Role: LEADER
I20260806 15:56:48.423967 3459 consensus_queue.cc:237] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63fcd4b64fa4052934d42805771d0a4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34027 } }
I20260806 15:56:48.424401 3453 sys_catalog.cc:564] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:48.425603 3467 sys_catalog.cc:455] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d63fcd4b64fa4052934d42805771d0a4. Latest consensus state: current_term: 1 leader_uuid: "d63fcd4b64fa4052934d42805771d0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63fcd4b64fa4052934d42805771d0a4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34027 } } }
I20260806 15:56:48.425735 3467 sys_catalog.cc:458] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:48.426149 3476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:48.426107 3462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d63fcd4b64fa4052934d42805771d0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63fcd4b64fa4052934d42805771d0a4" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 34027 } } }
I20260806 15:56:48.426291 3462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:48.429456 3476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:48.435804 3476 catalog_manager.cc:1347] Generated new cluster ID: 9ca7a32c695e43778838b1b48537cb1e
I20260806 15:56:48.435892 3476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:56:48.449499 3476 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:56:48.450480 3476 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:56:48.458328 3476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d63fcd4b64fa4052934d42805771d0a4: Generated new TSK 0
I20260806 15:56:48.459057 3476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:56:48.476541 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 3350
2026-08-06T15:56:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1655 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:56:48 dist-test-slave-rwrg krb5kdc[3496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:56:48 dist-test-slave-rwrg krb5kdc[3496](info): set up 2 sockets
Aug 06 15:56:48 dist-test-slave-rwrg krb5kdc[3496](info): commencing operation
krb5kdc: starting...
W20260806 15:56:49.751746 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:56:49 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:56:49.882884 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42535
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:49.993782 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:49.994066 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:49.994130 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:49.998013 3520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:49.998097 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:49.998132 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:49.998160 3520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:49.998198 3520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:50.003346 3520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:42535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:50.004390 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:50.005455 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:50.014050 3531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:50.014375 3520 server_base.cc:1034] running on GCE node
W20260806 15:56:50.014088 3527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.014088 3529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:50.015115 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:50.016260 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:50.017441 3520 hybrid_clock.cc:648] HybridClock initialized: now 1786031810017414 us; error 36 us; skew 500 ppm
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:50.020231 3520 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:50.021221 3520 webserver.cc:460] Webserver started at http://127.13.223.62:33867/ using document root <none> and password file <none>
I20260806 15:56:50.021806 3520 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:50.021896 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:50.022133 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:50.024135 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "2cedee238cb532e19fc244141fba7360"
server_key_iv: "01e9d2656222dd9649fec8e34bd016a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.024757 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "2cedee238cb532e19fc244141fba7360"
server_key_iv: "01e9d2656222dd9649fec8e34bd016a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.028644 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:56:50.031340 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.032742 3520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:56:50.032878 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "2cedee238cb532e19fc244141fba7360"
server_key_iv: "01e9d2656222dd9649fec8e34bd016a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.032991 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:50.057492 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:50.060554 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:50.060757 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:50.068434 3520 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:42535
I20260806 15:56:50.068790 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:42535 every 8 connection(s)
I20260806 15:56:50.069525 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:50.072299 3635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:50.078064 3635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap starting.
I20260806 15:56:50.079528 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 3520
I20260806 15:56:50.079680 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:56:50.079962 14204 external_mini_cluster.cc:1428] Setting key 06c7c409a69f18cbb5e86e3e3590594a
I20260806 15:56:50.080989 3635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:50.081773 3635 log.cc:824] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:50.083978 3635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: No bootstrap required, opened a new log
I20260806 15:56:50.088461 3635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:50.088820 3635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:50.088881 3635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b17d2687d4133b7ac2ec0c9c0cec7, State: Initialized, Role: FOLLOWER
I20260806 15:56:50.089423 3635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:50.089555 3635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:56:50.089639 3635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:56:50.089759 3635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:50.090906 3635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:50.091334 3635 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0b17d2687d4133b7ac2ec0c9c0cec7; no voters:
I20260806 15:56:50.091748 3635 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:56:50.091902 3648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:50.092147 3648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0b17d2687d4133b7ac2ec0c9c0cec7, State: Running, Role: LEADER
I20260806 15:56:50.092465 3648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:50.093259 3635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:50.094163 3650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } } }
I20260806 15:56:50.094304 3650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:50.094627 3649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0b17d2687d4133b7ac2ec0c9c0cec7. Latest consensus state: current_term: 1 leader_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } } }
I20260806 15:56:50.094719 3649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:50.095728 3663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:50.096895 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.081247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.13.223.62:42535)
0806 15:56:50.081788 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.081807 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.081850 (+ 43us) server_negotiation.cc:407] Connection header received
0806 15:56:50.082749 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.082770 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.083119 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.083394 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:50.084355 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.085563 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.086815 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.087305 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.087434 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.090458 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:50.090464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:50.090481 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:50.090516 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:50.092644 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.093184 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.093189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.093195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.093344 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.093675 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.093679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.093681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.094275 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:50.095320 (+ 1045us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.095472 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.095807 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":234,"threads_started":1}
I20260806 15:56:50.098899 3663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:50.105279 3663 catalog_manager.cc:1347] Generated new cluster ID: 41b43b3db36e4fe58842051b8af0c553
I20260806 15:56:50.105355 3663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:56:50.112193 3663 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:56:50.113152 3663 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:56:50.120285 3663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Generated new TSK 0
I20260806 15:56:50.120949 3663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:56:50.150931 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42535
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:50.269019 3678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:50.269279 3678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:50.269322 3678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:50.273183 3678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:50.273231 3678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:50.273284 3678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:56:50.278463 3678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:50.279438 3678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:50.280402 3678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:50.289875 3697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.289924 3686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.289922 3691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:50.290083 3678 server_base.cc:1034] running on GCE node
I20260806 15:56:50.290544 3678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:50.291175 3678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:50.292351 3678 hybrid_clock.cc:648] HybridClock initialized: now 1786031810292319 us; error 43 us; skew 500 ppm
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:50.295208 3678 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:56:50.296131 3678 webserver.cc:460] Webserver started at http://127.13.223.1:45041/ using document root <none> and password file <none>
I20260806 15:56:50.296748 3678 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:50.296835 3678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:50.297049 3678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:50.299100 3678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "357bdf25e1a24c61897f153c85fbaffa"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "e9a208a5e6d8aac76cb00b1861ddb9c7"
server_key_iv: "69ebe05b07ae7559eb0b4d095c4ace70"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.299679 3678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "357bdf25e1a24c61897f153c85fbaffa"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "e9a208a5e6d8aac76cb00b1861ddb9c7"
server_key_iv: "69ebe05b07ae7559eb0b4d095c4ace70"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.303424 3678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 15:56:50.305967 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.307144 3678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:56:50.307273 3678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "357bdf25e1a24c61897f153c85fbaffa"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "e9a208a5e6d8aac76cb00b1861ddb9c7"
server_key_iv: "69ebe05b07ae7559eb0b4d095c4ace70"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.307377 3678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:50.322443 3678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:50.325592 3678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:50.325811 3678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:50.326409 3678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:50.327251 3678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:50.327320 3678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.327390 3678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:50.327437 3678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.336750 3678 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:45737
I20260806 15:56:50.336804 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:45737 every 8 connection(s)
I20260806 15:56:50.337769 3678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:56:50.338232 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 3678
I20260806 15:56:50.338397 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:56:50.338757 14204 external_mini_cluster.cc:1428] Setting key c388228fccf280ed469a21324bf793ed
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:56:50.353559 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.340707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:50529 (local address 127.13.223.62:42535)
0806 15:56:50.340839 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.340846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.340901 (+ 55us) server_negotiation.cc:407] Connection header received
0806 15:56:50.342386 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.342390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.342457 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.342575 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:50.343932 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.344468 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.346406 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.346646 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.346734 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.349754 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:50.349757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:50.349760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:50.349786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:50.351303 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.351837 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.351841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.351843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.351928 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.352282 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.352286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.352288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.352477 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:50.352642 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.353275 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.353429 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 15:56:50.354842 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.340168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:50529)
0806 15:56:50.340753 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.340810 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.342157 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.342767 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.342775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.343157 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:50.343762 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.343785 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.344650 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.344657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.346263 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.346280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.346824 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.346831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.347111 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.347800 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:50.347825 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:50.349569 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:50.351428 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:50.351434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:50.351450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:50.351731 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:50.352021 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:50.352025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:50.352027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:50.352184 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:50.352648 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:50.352653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:50.352966 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.353202 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.353640 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":150,"threads_started":1}
I20260806 15:56:50.356279 3901 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42535
I20260806 15:56:50.356581 3901 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:50.357192 3901 heartbeater.cc:511] Master 127.13.223.62:42535 requested a full tablet report, sending...
I20260806 15:56:50.358865 3580 ts_manager.cc:194] Registered new tserver with Master: 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737)
I20260806 15:56:50.360008 3580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:50529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:56:50.371384 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42535
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:50.485761 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:50.486102 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:50.486179 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:50.491058 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:50.491148 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:50.491248 3908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:56:50.496940 3908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:50.498091 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:50.499142 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:50.508874 3917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.508857 3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.508885 3922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:50.509115 3908 server_base.cc:1034] running on GCE node
I20260806 15:56:50.509747 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:50.510411 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:50.511600 3908 hybrid_clock.cc:648] HybridClock initialized: now 1786031810511582 us; error 30 us; skew 500 ppm
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:50.514488 3908 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:56:50.515370 3908 webserver.cc:460] Webserver started at http://127.13.223.2:36323/ using document root <none> and password file <none>
I20260806 15:56:50.515928 3908 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:50.516013 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:50.516260 3908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:50.518405 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "178c7a5714dc4493a21b63c26ccbe723"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "88f9f8b77ed5d14b4cd83c07e88e8c80"
server_key_iv: "d6af9d1ebf566666f944e45487c08053"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.518955 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "178c7a5714dc4493a21b63c26ccbe723"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "88f9f8b77ed5d14b4cd83c07e88e8c80"
server_key_iv: "d6af9d1ebf566666f944e45487c08053"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.522637 3908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:56:50.525084 3931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.526270 3908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:56:50.526413 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "178c7a5714dc4493a21b63c26ccbe723"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "88f9f8b77ed5d14b4cd83c07e88e8c80"
server_key_iv: "d6af9d1ebf566666f944e45487c08053"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.526515 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:50.540125 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:50.543489 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:50.543699 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:50.544305 3908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:50.545243 3908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:50.545311 3908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.545379 3908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:50.545432 3908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.555593 3908 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:40787
I20260806 15:56:50.555613 4109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:40787 every 8 connection(s)
I20260806 15:56:50.556618 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:56:50.558917 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 3908
I20260806 15:56:50.559062 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:56:50.559410 14204 external_mini_cluster.cc:1428] Setting key a2d3d29d54fffb6166f2162dc2a4a6aa
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:56:50.572324 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.558502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:44477 (local address 127.13.223.62:42535)
0806 15:56:50.558642 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.558654 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.559612 (+ 958us) server_negotiation.cc:407] Connection header received
0806 15:56:50.561395 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.561399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.561454 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.561546 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:50.562863 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.563569 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.564881 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.565132 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.565251 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.568230 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:50.568233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:50.568235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:50.568259 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:50.569865 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.570434 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.570443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.570444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.570523 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.570873 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.570878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.570880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.571076 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:50.571181 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.571921 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.572089 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:56:50.573292 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.558882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:44477)
0806 15:56:50.559426 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.559497 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.561154 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.561760 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.561768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.562141 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:50.562692 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.562707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.563712 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.563716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.564751 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.564762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.565295 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.565299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.565487 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.566220 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:50.566237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:50.568038 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:50.570009 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:50.570015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:50.570031 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:50.570316 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:50.570625 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:50.570629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:50.570631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:50.570775 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:50.571196 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:50.571211 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:50.571572 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.571832 (+ 260us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.572252 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:56:50.574700 4111 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42535
I20260806 15:56:50.574960 4111 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:50.575492 4111 heartbeater.cc:511] Master 127.13.223.62:42535 requested a full tablet report, sending...
I20260806 15:56:50.576831 3580 ts_manager.cc:194] Registered new tserver with Master: 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
I20260806 15:56:50.577436 3580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:44477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:56:50.589926 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42535
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:50.703274 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:50.703553 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:50.703625 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:50.707732 4120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:50.707808 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:50.707886 4120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:56:50.713150 4120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:50.714128 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:50.715123 4120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:50.724985 4127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.725049 4126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:50.724989 4129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:50.725224 4120 server_base.cc:1034] running on GCE node
I20260806 15:56:50.725850 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:50.726512 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:50.727696 4120 hybrid_clock.cc:648] HybridClock initialized: now 1786031810727679 us; error 34 us; skew 500 ppm
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:50.730717 4120 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:56:50.731573 4120 webserver.cc:460] Webserver started at http://127.13.223.3:35611/ using document root <none> and password file <none>
I20260806 15:56:50.732125 4120 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:50.732211 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:50.732434 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:56:50.734418 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "457ce5cfcabd443f9862ade2e925989a"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "83f7e2ae4d8df0fb1a841f9e9f868304"
server_key_iv: "9c660fd76eff17d319ce3795c70461a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.735008 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "457ce5cfcabd443f9862ade2e925989a"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "83f7e2ae4d8df0fb1a841f9e9f868304"
server_key_iv: "9c660fd76eff17d319ce3795c70461a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.738893 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:56:50.741411 4146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.742677 4120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:56:50.742810 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "457ce5cfcabd443f9862ade2e925989a"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "83f7e2ae4d8df0fb1a841f9e9f868304"
server_key_iv: "9c660fd76eff17d319ce3795c70461a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:50.742931 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:50.778981 4120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:50.782063 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:50.782294 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:50.782941 4120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:50.783861 4120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:50.783929 4120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.783999 4120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:50.784042 4120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:50.793524 4120 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:44209
I20260806 15:56:50.793890 4310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:44209 every 8 connection(s)
I20260806 15:56:50.794766 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:56:50.797303 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 4120
I20260806 15:56:50.797443 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:56:50.797804 14204 external_mini_cluster.cc:1428] Setting key a9ddc88467a7dad130ae35b4b5aca92e
Aug 06 15:56:50 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:50.809000 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.796738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:41325 (local address 127.13.223.62:42535)
0806 15:56:50.796894 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.796898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.797803 (+ 905us) server_negotiation.cc:407] Connection header received
0806 15:56:50.799473 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.799477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.799569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.799672 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:50.800917 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.801441 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.802799 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.803023 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.803116 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.805649 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:50.805652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:50.805654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:50.805678 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:50.806943 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.807460 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.807467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.807470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.807553 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.807911 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.807914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.807916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.808090 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:50.808194 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.808715 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.808833 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 15:56:50.809814 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.797130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:41325)
0806 15:56:50.797616 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.797670 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.799283 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.799818 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.799824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.800295 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:50.800785 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.800799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.801644 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.801647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.802672 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.802682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.803163 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.803166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.803305 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.803947 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:50.803964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:50.805450 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:50.807063 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:50.807068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:50.807083 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:50.807347 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:50.807654 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:50.807657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:50.807659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:50.807796 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:50.808192 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:50.808199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:50.808435 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.808657 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.808974 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260806 15:56:50.811267 4311 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42535
I20260806 15:56:50.811530 4311 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:50.812127 4311 heartbeater.cc:511] Master 127.13.223.62:42535 requested a full tablet report, sending...
I20260806 15:56:50.813199 3580 ts_manager.cc:194] Registered new tserver with Master: 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209)
I20260806 15:56:50.813726 3580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:41325
I20260806 15:56:50.823738 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:56:50.833078 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.825821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.13.223.62:42535)
0806 15:56:50.825989 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.825992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.826126 (+ 134us) server_negotiation.cc:407] Connection header received
0806 15:56:50.826234 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.826236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.826277 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.826343 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:50.827015 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.827510 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.828310 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.828568 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.828673 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.830162 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:50.830165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:50.830167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:50.830187 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:50.831472 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.831892 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.831895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.831897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.831983 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:50.832202 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:50.832205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:50.832206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:50.832405 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:50.832554 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.832790 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.832880 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 15:56:50.835994 3580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:56:50.839165 3580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:56:50.853856 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.848975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:44209 (local address 127.0.0.1:36116)
0806 15:56:50.849503 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.849517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.849615 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.850034 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.850037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.850053 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:50.850162 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.850169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.851204 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.851210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.852455 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.852465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.853307 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.853312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.853422 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.853464 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.853589 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.853703 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":102,"threads_started":1}
I20260806 15:56:50.853892 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.849137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.13.223.3:44209)
0806 15:56:50.849434 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.849445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.849705 (+ 260us) server_negotiation.cc:407] Connection header received
0806 15:56:50.849772 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.849776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.849905 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.850055 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:50.850306 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.851071 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.852618 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.853191 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.853312 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.853465 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.853640 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.853737 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
I20260806 15:56:50.854562 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.847940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45737 (local address 127.0.0.1:40342)
0806 15:56:50.848743 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.848756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.848837 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.850105 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.850108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.850128 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:50.850222 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.850230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.851662 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.851665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.853082 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.853094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.854187 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.854191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.854295 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.854321 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.854372 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.854437 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":132,"thread_start_us":115,"threads_started":1}
I20260806 15:56:50.855290 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.848416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.13.223.1:45737)
0806 15:56:50.849732 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.849747 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.849768 (+ 21us) server_negotiation.cc:407] Connection header received
0806 15:56:50.849836 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.849839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.849974 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.850120 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:50.850335 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.851496 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.853220 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.854034 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.854836 (+ 802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.854991 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.855044 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.855144 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1193,"thread_start_us":134,"threads_started":1}
I20260806 15:56:50.855434 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.847849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:40787 (local address 127.0.0.1:44408)
0806 15:56:50.848307 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.848354 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.848484 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.849441 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.849447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.849474 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:50.849635 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.849645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.851662 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.851665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.853084 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.853094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.854953 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.854958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.855077 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.855108 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.855173 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.855258 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":84,"thread_start_us":94,"threads_started":1}
I20260806 15:56:50.856791 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.847993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.13.223.2:40787)
0806 15:56:50.848915 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.848929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.848953 (+ 24us) server_negotiation.cc:407] Connection header received
0806 15:56:50.849039 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.849047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.849284 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.850239 (+ 955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:50.850410 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.851512 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.853250 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.854591 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.856238 (+ 1647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.856398 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.856458 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.856606 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":159,"threads_started":1}
I20260806 15:56:50.857206 4216 tablet_service.cc:1467] Processing CreateTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 (DEFAULT_TABLE table=test-table [id=8cf27fd52e1d41ccad109b7e4f5ca304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:50.858304 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e521e8f3744b1a31dc04a74310cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:50.858261 3791 tablet_service.cc:1467] Processing CreateTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 (DEFAULT_TABLE table=test-table [id=8cf27fd52e1d41ccad109b7e4f5ca304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:50.859375 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e521e8f3744b1a31dc04a74310cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:50.859465 4008 tablet_service.cc:1467] Processing CreateTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 (DEFAULT_TABLE table=test-table [id=8cf27fd52e1d41ccad109b7e4f5ca304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:50.860369 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e521e8f3744b1a31dc04a74310cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:50.865839 4344 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap starting.
I20260806 15:56:50.867074 4347 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Bootstrap starting.
I20260806 15:56:50.867672 4345 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Bootstrap starting.
I20260806 15:56:50.868634 4344 tablet_bootstrap.cc:654] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:50.869549 4344 log.cc:824] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:50.869715 4347 tablet_bootstrap.cc:654] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:50.869827 4345 tablet_bootstrap.cc:654] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:50.870647 4347 log.cc:824] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:50.871062 4345 log.cc:824] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:50.872047 4344 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: No bootstrap required, opened a new log
I20260806 15:56:50.872313 4344 ts_tablet_manager.cc:1397] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260806 15:56:50.872655 4347 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: No bootstrap required, opened a new log
I20260806 15:56:50.872838 4347 ts_tablet_manager.cc:1397] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:56:50.873400 4345 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: No bootstrap required, opened a new log
I20260806 15:56:50.873610 4345 ts_tablet_manager.cc:1397] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:56:50.875660 4347 raft_consensus.cc:348] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.875895 4347 raft_consensus.cc:374] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:50.875954 4347 raft_consensus.cc:729] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178c7a5714dc4493a21b63c26ccbe723, State: Initialized, Role: FOLLOWER
I20260806 15:56:50.876417 4347 consensus_queue.cc:260] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.877097 4111 heartbeater.cc:503] Master 127.13.223.62:42535 was elected leader, sending a full tablet report...
I20260806 15:56:50.877449 4347 ts_tablet_manager.cc:1428] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 15:56:50.877485 4344 raft_consensus.cc:348] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.877790 4344 raft_consensus.cc:374] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:50.877851 4344 raft_consensus.cc:729] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357bdf25e1a24c61897f153c85fbaffa, State: Initialized, Role: FOLLOWER
I20260806 15:56:50.878345 4344 consensus_queue.cc:260] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.879096 3901 heartbeater.cc:503] Master 127.13.223.62:42535 was elected leader, sending a full tablet report...
I20260806 15:56:50.879120 4345 raft_consensus.cc:348] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.879387 4345 raft_consensus.cc:374] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:50.879456 4345 raft_consensus.cc:729] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457ce5cfcabd443f9862ade2e925989a, State: Initialized, Role: FOLLOWER
I20260806 15:56:50.879889 4345 consensus_queue.cc:260] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.880867 4344 ts_tablet_manager.cc:1428] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 15:56:50.880956 4345 ts_tablet_manager.cc:1428] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 15:56:50.881318 4311 heartbeater.cc:503] Master 127.13.223.62:42535 was elected leader, sending a full tablet report...
I20260806 15:56:50.988673 4354 raft_consensus.cc:482] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:50.988943 4354 raft_consensus.cc:504] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.990291 4354 leader_election.cc:288] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737), 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209)
I20260806 15:56:50.994328 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.990678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:34977 (local address 127.13.223.1:45737)
0806 15:56:50.990843 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.990849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.990900 (+ 51us) server_negotiation.cc:407] Connection header received
0806 15:56:50.990967 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.990971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.991034 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.991132 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:50.991459 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.991990 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.993293 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.993724 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.993857 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.993912 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.994103 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.994205 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 15:56:50.994331 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.990561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45737 (local address 127.13.223.2:34977)
0806 15:56:50.990709 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.990727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.990832 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.991149 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.991152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.991172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:50.991317 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.991323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.992119 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.992125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.993167 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.993178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.993849 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.993853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.993972 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.993990 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.994084 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.994160 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":88,"spinlock_wait_cycles":23296}
I20260806 15:56:50.994508 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.990841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:60067 (local address 127.13.223.3:44209)
0806 15:56:50.990981 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:50.990985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:50.991179 (+ 194us) server_negotiation.cc:407] Connection header received
0806 15:56:50.991341 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:50.991357 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:50.991401 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:50.991482 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:50.991737 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.992223 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.993429 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.993822 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.993943 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.993994 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:50.994279 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 15:56:50.994351 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:56:50.994659 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:50.990627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:44209 (local address 127.13.223.2:60067)
0806 15:56:50.991080 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:50.991090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:50.991236 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:50.991494 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:50.991497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:50.991512 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:50.991630 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.991635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.992335 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.992340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.993313 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:50.993323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:50.993943 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:50.993946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:50.994060 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:50.994077 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 15:56:50.994252 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 15:56:50.994323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":158,"threads_started":1}
I20260806 15:56:50.995139 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e521e8f3744b1a31dc04a74310cf2" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357bdf25e1a24c61897f153c85fbaffa" is_pre_election: true
I20260806 15:56:50.995360 4245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e521e8f3744b1a31dc04a74310cf2" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457ce5cfcabd443f9862ade2e925989a" is_pre_election: true
I20260806 15:56:50.995569 3826 raft_consensus.cc:2454] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178c7a5714dc4493a21b63c26ccbe723 in term 0.
I20260806 15:56:50.995626 4245 raft_consensus.cc:2454] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178c7a5714dc4493a21b63c26ccbe723 in term 0.
I20260806 15:56:50.996110 3932 leader_election.cc:302] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178c7a5714dc4493a21b63c26ccbe723, 457ce5cfcabd443f9862ade2e925989a; no voters:
I20260806 15:56:50.996366 4354 raft_consensus.cc:2781] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:56:50.996449 4354 raft_consensus.cc:482] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:56:50.996479 4354 raft_consensus.cc:3037] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:50.997539 4354 raft_consensus.cc:504] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:50.997900 4354 leader_election.cc:288] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 election: Requested vote from peers 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737), 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209)
I20260806 15:56:50.998270 4245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e521e8f3744b1a31dc04a74310cf2" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457ce5cfcabd443f9862ade2e925989a"
I20260806 15:56:50.998330 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e521e8f3744b1a31dc04a74310cf2" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357bdf25e1a24c61897f153c85fbaffa"
I20260806 15:56:50.998399 4245 raft_consensus.cc:3037] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:50.998493 3826 raft_consensus.cc:3037] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:50.999491 4245 raft_consensus.cc:2454] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178c7a5714dc4493a21b63c26ccbe723 in term 1.
I20260806 15:56:50.999786 3826 raft_consensus.cc:2454] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178c7a5714dc4493a21b63c26ccbe723 in term 1.
I20260806 15:56:50.999818 3932 leader_election.cc:302] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178c7a5714dc4493a21b63c26ccbe723, 457ce5cfcabd443f9862ade2e925989a; no voters:
I20260806 15:56:51.000072 4354 raft_consensus.cc:2781] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:51.000337 4354 raft_consensus.cc:686] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 LEADER]: Becoming Leader. State: Replica: 178c7a5714dc4493a21b63c26ccbe723, State: Running, Role: LEADER
I20260806 15:56:51.000864 4354 consensus_queue.cc:237] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:51.004238 3580 catalog_manager.cc:5495] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 reported cstate change: term changed from 0 to 1, leader changed from <none> to 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "178c7a5714dc4493a21b63c26ccbe723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } health_report { overall_health: UNKNOWN } } }
I20260806 15:56:51.044768 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.040811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.13.223.2:40787)
0806 15:56:51.040973 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:51.040979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:51.040995 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:56:51.041090 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:51.041092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:51.041157 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:51.041281 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:56:51.041548 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.042096 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.043071 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.043312 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.043429 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.043777 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:56:51.044241 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:56:51.044383 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:51.044457 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 15:56:51.044596 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 15:56:51.045872 4312 tablet.cc:2369] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:51.051460 4245 raft_consensus.cc:1261] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Refusing update from remote peer 178c7a5714dc4493a21b63c26ccbe723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:51.051460 3826 raft_consensus.cc:1261] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Refusing update from remote peer 178c7a5714dc4493a21b63c26ccbe723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:51.052181 4354 consensus_queue.cc:1035] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:51.052405 4361 consensus_queue.cc:1035] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 15:56:51.057859 4113 tablet.cc:2369] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:51.062314 4366 mvcc.cc:205] Tried to move back new op lower bound from 7315586298059096064 to 7315586297862897664. Current Snapshot: MvccSnapshot[applied={T|T < 7315586298059096064}]
I20260806 15:56:51.062938 4367 mvcc.cc:205] Tried to move back new op lower bound from 7315586298059096064 to 7315586297862897664. Current Snapshot: MvccSnapshot[applied={T|T < 7315586298059096064}]
I20260806 15:56:51.070264 3580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:56:51.070446 3580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:56:51.072949 3580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Sending DeleteTablet for 3 replicas of tablet ee8e521e8f3744b1a31dc04a74310cf2
I20260806 15:56:51.073839 4008 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:44408
I20260806 15:56:51.073925 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40342
I20260806 15:56:51.074384 4216 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260806 15:56:51.074892 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 3678
I20260806 15:56:51.088766 4378 tablet_replica.cc:330] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: stopping tablet replica
I20260806 15:56:51.089183 4378 raft_consensus.cc:2229] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:56:51.089592 4378 raft_consensus.cc:2258] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:51.095232 4378 ts_tablet_manager.cc:1905] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:51.095718 4376 tablet_replica.cc:330] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: stopping tablet replica
I20260806 15:56:51.096130 4376 raft_consensus.cc:2229] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:56:51.096657 4376 raft_consensus.cc:2258] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:51.098572 4376 ts_tablet_manager.cc:1905] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:51.099669 4378 ts_tablet_manager.cc:1918] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:51.099792 4378 log.cc:1196] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/ee8e521e8f3744b1a31dc04a74310cf2
I20260806 15:56:51.100198 4378 ts_tablet_manager.cc:1939] T ee8e521e8f3744b1a31dc04a74310cf2 P 457ce5cfcabd443f9862ade2e925989a: Deleting consensus metadata
I20260806 15:56:51.101686 4376 ts_tablet_manager.cc:1918] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:51.101727 3551 catalog_manager.cc:4841] TS 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209): tablet ee8e521e8f3744b1a31dc04a74310cf2 (table test-table [id=8cf27fd52e1d41ccad109b7e4f5ca304]) successfully deleted
I20260806 15:56:51.101912 4376 log.cc:1196] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/ee8e521e8f3744b1a31dc04a74310cf2
I20260806 15:56:51.102329 4376 ts_tablet_manager.cc:1939] T ee8e521e8f3744b1a31dc04a74310cf2 P 178c7a5714dc4493a21b63c26ccbe723: Deleting consensus metadata
I20260806 15:56:51.103609 3552 catalog_manager.cc:4841] TS 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787): tablet ee8e521e8f3744b1a31dc04a74310cf2 (table test-table [id=8cf27fd52e1d41ccad109b7e4f5ca304]) successfully deleted
W20260806 15:56:51.105015 3557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:45737 (error 108)
I20260806 15:56:51.105916 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 3908
I20260806 15:56:51.106257 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.105902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45737 (local address 127.0.0.1:40358)
0806 15:56:51.106075 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:51.106157 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:45737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 15:56:51.125236 3557 catalog_manager.cc:4584] TS 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee8e521e8f3744b1a31dc04a74310cf2: Network error: Client connection negotiation failed: client connection to 127.13.223.1:45737: connect: Connection refused (error 111)
I20260806 15:56:51.135262 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 4120
I20260806 15:56:51.147070 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 3520
I20260806 15:56:51.159660 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42535
--webserver_interface=127.13.223.62
--webserver_port=33867
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:51.273821 4380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:51.274052 4380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:51.274092 4380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:51.278050 4380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:51.278103 4380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:51.278121 4380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:51.278138 4380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:51.278153 4380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:51.283222 4380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:42535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:51.284200 4380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:51.285238 4380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:51.298918 4393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.299140 4388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.299265 4387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:51.299635 4380 server_base.cc:1034] running on GCE node
I20260806 15:56:51.300206 4380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:51.301354 4380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:51.302532 4380 hybrid_clock.cc:648] HybridClock initialized: now 1786031811302498 us; error 59 us; skew 500 ppm
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:51.305366 4380 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:51.306255 4380 webserver.cc:460] Webserver started at http://127.13.223.62:33867/ using document root <none> and password file <none>
I20260806 15:56:51.306814 4380 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:51.306926 4380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:51.310783 4380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 15:56:51.313122 4403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.314369 4380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:56:51.314530 4380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "2cedee238cb532e19fc244141fba7360"
server_key_iv: "01e9d2656222dd9649fec8e34bd016a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:51.315200 4380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:51.349061 4380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:51.352982 4380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:51.353248 4380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:51.360679 4380 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:42535
I20260806 15:56:51.360976 4489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:42535 every 8 connection(s)
I20260806 15:56:51.361929 4380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:51.366532 4490 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:56:51.366955 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 4380
I20260806 15:56:51.367424 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:45737
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:42535
--webserver_port=45041
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:51.368443 4490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap starting.
I20260806 15:56:51.393397 4490 log.cc:824] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:51.405730 4490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:56:51.406217 4490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap complete.
I20260806 15:56:51.411983 4490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:51.412585 4490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b17d2687d4133b7ac2ec0c9c0cec7, State: Initialized, Role: FOLLOWER
I20260806 15:56:51.413205 4490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:51.413400 4490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:56:51.413497 4490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:56:51.413607 4490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 15:56:51.416306 4490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:51.416742 4490 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0b17d2687d4133b7ac2ec0c9c0cec7; no voters:
I20260806 15:56:51.417125 4490 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 15:56:51.417258 4494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 15:56:51.417548 4494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 LEADER]: Becoming Leader. State: Replica: 3d0b17d2687d4133b7ac2ec0c9c0cec7, State: Running, Role: LEADER
I20260806 15:56:51.417850 4494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:51.418435 4490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:51.419205 4499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0b17d2687d4133b7ac2ec0c9c0cec7. Latest consensus state: current_term: 2 leader_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } } }
I20260806 15:56:51.419335 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } } }
I20260806 15:56:51.419410 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:51.419345 4499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:51.421108 4510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:51.425419 4510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:51.425935 4510 catalog_manager.cc:1259] Loaded cluster ID: 41b43b3db36e4fe58842051b8af0c553
I20260806 15:56:51.426010 4510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:56:51.427109 4510 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 15:56:51.429839 4510 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 15:56:51.434388 4510 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:56:51.435024 4510 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Loaded TSK: 0
I20260806 15:56:51.436013 4510 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 15:56:51.521080 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:51.521335 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:51.521378 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:51.525489 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:51.525557 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:51.525645 4492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:56:51.530864 4492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:45737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=45041
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:51.531880 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:51.532944 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:51.542734 4538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.542735 4537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.542747 4541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:51.543813 4492 server_base.cc:1034] running on GCE node
I20260806 15:56:51.544225 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:51.545082 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:51.546274 4492 hybrid_clock.cc:648] HybridClock initialized: now 1786031811546251 us; error 42 us; skew 500 ppm
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:51.549151 4492 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:56:51.550177 4492 webserver.cc:460] Webserver started at http://127.13.223.1:45041/ using document root <none> and password file <none>
I20260806 15:56:51.550741 4492 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:51.550818 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:51.555487 4492 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:56:51.558017 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.559505 4492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:56:51.559612 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "357bdf25e1a24c61897f153c85fbaffa"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "e9a208a5e6d8aac76cb00b1861ddb9c7"
server_key_iv: "69ebe05b07ae7559eb0b4d095c4ace70"
server_key_version: "encryptionkey@0"
I20260806 15:56:51.560231 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:51.572181 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:51.575845 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:51.576092 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:51.576881 4492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:51.578158 4572 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 15:56:51.582712 4492 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 15:56:51.582782 4492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260806 15:56:51.582854 4492 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 15:56:51.584745 4492 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 15:56:51.584823 4492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 15:56:51.585099 4572 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap starting.
I20260806 15:56:51.597440 4492 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:45737
I20260806 15:56:51.597494 4746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:45737 every 8 connection(s)
I20260806 15:56:51.598894 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:56:51.606381 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 4492
I20260806 15:56:51.606845 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:40787
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:42535
--webserver_port=36323
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:51.609683 4572 log.cc:824] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Log is configured to *not* fsync() on all Append() calls
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:51.617576 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.601332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:34249)
0806 15:56:51.601930 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:51.601972 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:51.602887 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:51.604000 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:51.604009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:51.604453 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:51.604996 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.605011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.606468 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.606476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.608086 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.608099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.609128 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.609133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.609363 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.609987 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:51.610004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:51.611691 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:51.614177 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.614184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.614200 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.614492 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.614900 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.614904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.614907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.615069 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.615690 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:51.615696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:51.615960 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 15:56:51.616171 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 15:56:51.616578 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":164,"threads_started":1}
I20260806 15:56:51.617573 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.601286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:34249 (local address 127.13.223.62:42535)
0806 15:56:51.601661 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:51.601677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:51.602114 (+ 437us) server_negotiation.cc:407] Connection header received
0806 15:56:51.603311 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:51.603321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:51.603616 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:51.603940 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:51.605193 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.606252 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.608257 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.608961 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.609159 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.611935 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:51.611941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:51.611955 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:51.611987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:51.614031 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.614649 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.614655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.614659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.614789 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.615179 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.615184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.615186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.615551 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:51.615720 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:51.616297 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 15:56:51.616621 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":128,"threads_started":1}
I20260806 15:56:51.620061 4572 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:56:51.620574 4572 tablet_bootstrap.cc:492] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap complete.
I20260806 15:56:51.620896 4747 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42535
I20260806 15:56:51.621145 4747 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:51.621685 4747 heartbeater.cc:511] Master 127.13.223.62:42535 requested a full tablet report, sending...
I20260806 15:56:51.622231 4572 ts_tablet_manager.cc:1397] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.011s
I20260806 15:56:51.624567 4435 ts_manager.cc:194] Registered new tserver with Master: 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737)
I20260806 15:56:51.625221 4572 raft_consensus.cc:348] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:51.625742 4572 raft_consensus.cc:729] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357bdf25e1a24c61897f153c85fbaffa, State: Initialized, Role: FOLLOWER
I20260806 15:56:51.626259 4572 consensus_queue.cc:260] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } } peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } }
I20260806 15:56:51.627261 4572 ts_tablet_manager.cc:1428] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260806 15:56:51.628912 4435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:34249
I20260806 15:56:51.631714 4747 heartbeater.cc:503] Master 127.13.223.62:42535 was elected leader, sending a full tablet report...
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.1@KRBTEST.COM
I20260806 15:56:51.639550 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.628077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45737 (local address 127.0.0.1:40366)
0806 15:56:51.628559 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:51.628641 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:51.628794 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:51.629655 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:51.629664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:51.630159 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:51.630267 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.630275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.631374 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.631379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.632244 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.632257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.633088 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.633093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.633212 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.633950 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:51.633963 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:51.635480 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:51.637758 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.637768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.637774 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.638020 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.638351 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.638354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.638355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.638451 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.638981 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:51.638987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:51.639117 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 15:56:51.639244 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 15:56:51.639407 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
I20260806 15:56:51.639693 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.628831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.13.223.1:45737)
0806 15:56:51.629162 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:51.629174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:51.629196 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:56:51.629276 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:51.629281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:51.629469 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:51.629615 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:51.630407 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.631232 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.632374 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.632936 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.633047 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.635639 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:51.635648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:51.635662 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:51.635688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:51.637637 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.638149 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.638155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.638157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.638254 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.638543 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.638546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.638548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.638853 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:51.638985 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:51.639348 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 15:56:51.639522 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":134,"threads_started":1}
I20260806 15:56:51.640974 4635 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8e521e8f3744b1a31dc04a74310cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:51 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40366
I20260806 15:56:51.641997 4762 tablet_replica.cc:330] stopping tablet replica
I20260806 15:56:51.642323 4762 raft_consensus.cc:2229] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:56:51.642593 4762 raft_consensus.cc:2258] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:51.643771 4762 ts_tablet_manager.cc:1905] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:51.647447 4762 ts_tablet_manager.cc:1918] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:51.647545 4762 log.cc:1196] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/ee8e521e8f3744b1a31dc04a74310cf2
I20260806 15:56:51.647854 4762 ts_tablet_manager.cc:1939] T ee8e521e8f3744b1a31dc04a74310cf2 P 357bdf25e1a24c61897f153c85fbaffa: Deleting consensus metadata
I20260806 15:56:51.649681 4410 catalog_manager.cc:4841] TS 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737): tablet ee8e521e8f3744b1a31dc04a74310cf2 (table test-table [id=8cf27fd52e1d41ccad109b7e4f5ca304]) successfully deleted
W20260806 15:56:51.734694 4753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:51.734979 4753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:51.735055 4753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:51.739101 4753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:51.739183 4753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:51.739269 4753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:56:51.744477 4753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:40787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=36323
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:51.745566 4753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:51.746584 4753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:51.756474 4768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.756564 4770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:51.756618 4753 server_base.cc:1034] running on GCE node
W20260806 15:56:51.756564 4767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:51.757110 4753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:51.757797 4753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:51.758967 4753 hybrid_clock.cc:648] HybridClock initialized: now 1786031811758954 us; error 34 us; skew 500 ppm
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:51.761901 4753 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:56:51.762743 4753 webserver.cc:460] Webserver started at http://127.13.223.2:36323/ using document root <none> and password file <none>
I20260806 15:56:51.763330 4753 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:51.763475 4753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:51.767531 4753 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 15:56:51.769603 4783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.770814 4753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:56:51.770906 4753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "178c7a5714dc4493a21b63c26ccbe723"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "88f9f8b77ed5d14b4cd83c07e88e8c80"
server_key_iv: "d6af9d1ebf566666f944e45487c08053"
server_key_version: "encryptionkey@0"
I20260806 15:56:51.771400 4753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:51.788810 4753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:51.791632 4753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:51.791788 4753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:51.792326 4753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:51.793223 4753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:51.793269 4753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.793311 4753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:51.793332 4753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.802495 4753 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:40787
I20260806 15:56:51.802515 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:40787 every 8 connection(s)
I20260806 15:56:51.803570 4753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:56:51.806020 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 4753
I20260806 15:56:51.806439 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:44209
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:42535
--webserver_port=35611
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:51.819712 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.805715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:59623 (local address 127.13.223.62:42535)
0806 15:56:51.805905 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:51.805910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:51.806862 (+ 952us) server_negotiation.cc:407] Connection header received
0806 15:56:51.808336 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:51.808340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:51.808416 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:51.808560 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:51.809942 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.810834 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.812070 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.812582 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.812736 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.815484 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:51.815490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:51.815497 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:51.815531 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:51.817410 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.817929 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.817934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.817938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.818040 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.818389 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.818394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.818398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.818698 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:51.818806 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:51.819342 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 15:56:51.819498 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 15:56:51.820638 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.806144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:59623)
0806 15:56:51.806675 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:51.806731 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:51.808087 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:51.808737 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:51.808744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:51.809147 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:51.809758 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.809785 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.810980 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.810983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.811933 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.811944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.812762 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.812768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.812943 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.813569 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:51.813585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:51.815301 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:51.817531 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.817536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.817548 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.817807 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.818152 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.818154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.818155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.818288 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.818820 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:51.818825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:51.819082 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 15:56:51.819250 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 15:56:51.819599 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":150,"threads_started":1}
I20260806 15:56:51.821921 4960 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42535
I20260806 15:56:51.822163 4960 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:51.822655 4960 heartbeater.cc:511] Master 127.13.223.62:42535 requested a full tablet report, sending...
I20260806 15:56:51.823756 4435 ts_manager.cc:194] Registered new tserver with Master: 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
I20260806 15:56:51.824561 4435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:59623
W20260806 15:56:51.922904 4963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:51.923151 4963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:51.923192 4963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:51.927064 4963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:51.927121 4963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:51.927176 4963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:56:51.932238 4963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:44209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=35611
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:51.933352 4963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:51.934398 4963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:51.943915 4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.943966 4973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:51.943912 4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:51.944258 4963 server_base.cc:1034] running on GCE node
I20260806 15:56:51.944763 4963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:51.945426 4963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:51.946587 4963 hybrid_clock.cc:648] HybridClock initialized: now 1786031811946570 us; error 27 us; skew 500 ppm
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:51.949461 4963 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:56:51.950321 4963 webserver.cc:460] Webserver started at http://127.13.223.3:35611/ using document root <none> and password file <none>
I20260806 15:56:51.950876 4963 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:51.950984 4963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:51.954846 4963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:56:51.956874 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.958196 4963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:56:51.958329 4963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "457ce5cfcabd443f9862ade2e925989a"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "83f7e2ae4d8df0fb1a841f9e9f868304"
server_key_iv: "9c660fd76eff17d319ce3795c70461a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:51.958848 4963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:51.970665 4963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:51.973788 4963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:51.973979 4963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:51.974633 4963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:51.975472 4963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:51.975539 4963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.975606 4963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:51.975657 4963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:51.984151 4963 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:44209
I20260806 15:56:51.984195 5175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:44209 every 8 connection(s)
I20260806 15:56:51.985165 4963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
Aug 06 15:56:51 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031811, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:56:51.994619 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 4963
I20260806 15:56:51.998868 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.986920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:35713 (local address 127.13.223.62:42535)
0806 15:56:51.987069 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:51.987073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:51.987893 (+ 820us) server_negotiation.cc:407] Connection header received
0806 15:56:51.989225 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:51.989228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:51.989285 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:51.989361 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:51.990441 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.990947 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.992010 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.992233 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.992308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.994930 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:51.994933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:51.994935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:51.994958 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:51.996591 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.997168 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.997172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.997173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.997264 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:51.997621 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:51.997625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:51.997627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:51.997820 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:51.997904 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:51.998607 (+ 703us) server_negotiation.cc:299] Negotiation successful
0806 15:56:51.998700 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:56:51.999931 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.987314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:35713)
0806 15:56:51.987744 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:51.987785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:51.989036 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:51.989521 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:51.989557 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:51.989880 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:51.990316 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.990328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.991088 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.991091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.991896 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:51.991905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.992363 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.992366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:51.992629 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:51.993152 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:51.993167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:51.994749 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:51.996710 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.996716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.996731 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.997073 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.997356 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:51.997358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:51.997359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:51.997519 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:51.997933 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:51.997939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:51.998286 (+ 347us) client_negotiation.cc:769] Sending connection context
0806 15:56:51.998554 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 15:56:51.998883 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260806 15:56:52.001268 5178 heartbeater.cc:347] Connected to a master server at 127.13.223.62:42535
I20260806 15:56:52.001590 5178 heartbeater.cc:464] Registering TS with master...
I20260806 15:56:52.002156 5178 heartbeater.cc:511] Master 127.13.223.62:42535 requested a full tablet report, sending...
I20260806 15:56:52.003427 4435 ts_manager.cc:194] Registered new tserver with Master: 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209)
I20260806 15:56:52.004112 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:51.995285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.13.223.62:42535)
0806 15:56:51.995549 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:51.995553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:51.995759 (+ 206us) server_negotiation.cc:407] Connection header received
0806 15:56:51.995927 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:51.995931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:51.995976 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:51.996070 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:51.996919 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.997621 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.998812 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:51.999150 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:51.999230 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.000759 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:52.000763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:52.000765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:52.000793 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:52.002337 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:52.002793 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:52.002797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:52.002799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:52.002888 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:52.003215 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:52.003218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:52.003220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:52.003412 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:52.003637 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.003853 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.003964 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260806 15:56:52.004140 4435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:35713
I20260806 15:56:52.005904 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:56:52.010258 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.006538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.13.223.62:42535)
0806 15:56:52.006672 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.006676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.006859 (+ 183us) server_negotiation.cc:407] Connection header received
0806 15:56:52.007036 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.007039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.007130 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.007233 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:56:52.007467 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.007935 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.008863 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.009119 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.009226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.009579 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:56:52.009904 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:56:52.010045 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.010086 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.010139 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:56:52.010856 4435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:56:52.014223 4435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:56:52.022972 4635 tablet_service.cc:1467] Processing CreateTablet for tablet 36e321f6fc6841f181b00a9bb1da9488 (DEFAULT_TABLE table=test-table [id=90ddf758f6994a48ab89270b47a5c5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:52.023690 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e321f6fc6841f181b00a9bb1da9488. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:52.026784 4572 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap starting.
I20260806 15:56:52.027508 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.021435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:44209 (local address 127.0.0.1:36124)
0806 15:56:52.021610 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.021627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.021714 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.022791 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:52.022794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:52.022812 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:52.023018 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.023027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.024613 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.024617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.025820 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.025831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.027053 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.027057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.027160 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.027219 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 15:56:52.027275 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 15:56:52.027361 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":110}
I20260806 15:56:52.027552 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.021635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:40787 (local address 127.0.0.1:44420)
0806 15:56:52.021915 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.021927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.022022 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.022820 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:52.022824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:52.022844 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:52.023127 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.023132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.024838 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.024841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.026031 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.026041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.027233 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.027236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.027322 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.027340 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 15:56:52.027396 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 15:56:52.027450 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":154,"thread_start_us":141,"threads_started":1}
I20260806 15:56:52.027782 4572 tablet_bootstrap.cc:654] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:52.028172 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.021976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.13.223.2:40787)
0806 15:56:52.022289 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.022302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.022326 (+ 24us) server_negotiation.cc:407] Connection header received
0806 15:56:52.022403 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.022409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.022612 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.023501 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:52.023705 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.024693 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.026166 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.027109 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.027224 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.027374 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.027429 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.027523 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":121,"threads_started":1}
I20260806 15:56:52.028215 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.021903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.13.223.3:44209)
0806 15:56:52.022286 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.022300 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.022325 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:56:52.022390 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.022394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.022617 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.023127 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:52.023288 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.024380 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.025929 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.026919 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.027847 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.027991 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.028034 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.028107 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":198,"threads_started":1}
I20260806 15:56:52.029971 4572 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: No bootstrap required, opened a new log
I20260806 15:56:52.030043 4572 ts_tablet_manager.cc:1397] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 15:56:52.030428 4572 raft_consensus.cc:348] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.030514 4572 raft_consensus.cc:374] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:52.030540 4572 raft_consensus.cc:729] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357bdf25e1a24c61897f153c85fbaffa, State: Initialized, Role: FOLLOWER
I20260806 15:56:52.030679 4572 consensus_queue.cc:260] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.030947 4572 ts_tablet_manager.cc:1428] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 15:56:52.031821 5077 tablet_service.cc:1467] Processing CreateTablet for tablet 36e321f6fc6841f181b00a9bb1da9488 (DEFAULT_TABLE table=test-table [id=90ddf758f6994a48ab89270b47a5c5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:52.031821 4852 tablet_service.cc:1467] Processing CreateTablet for tablet 36e321f6fc6841f181b00a9bb1da9488 (DEFAULT_TABLE table=test-table [id=90ddf758f6994a48ab89270b47a5c5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:56:52.033023 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e321f6fc6841f181b00a9bb1da9488. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:52.033028 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e321f6fc6841f181b00a9bb1da9488. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:56:52.041563 5196 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Bootstrap starting.
I20260806 15:56:52.041563 5195 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Bootstrap starting.
I20260806 15:56:52.044123 5196 tablet_bootstrap.cc:654] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:52.044685 5195 tablet_bootstrap.cc:654] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Neither blocks nor log segments found. Creating new log.
I20260806 15:56:52.045558 5196 log.cc:824] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:52.045558 5195 log.cc:824] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:52.047799 5196 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: No bootstrap required, opened a new log
I20260806 15:56:52.047799 5195 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: No bootstrap required, opened a new log
I20260806 15:56:52.048002 5196 ts_tablet_manager.cc:1397] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 15:56:52.048002 5195 ts_tablet_manager.cc:1397] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 15:56:52.052057 5196 raft_consensus.cc:348] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.052057 5195 raft_consensus.cc:348] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.052338 5196 raft_consensus.cc:374] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:52.052338 5195 raft_consensus.cc:374] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:56:52.052393 5196 raft_consensus.cc:729] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457ce5cfcabd443f9862ade2e925989a, State: Initialized, Role: FOLLOWER
I20260806 15:56:52.052393 5195 raft_consensus.cc:729] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178c7a5714dc4493a21b63c26ccbe723, State: Initialized, Role: FOLLOWER
I20260806 15:56:52.052902 5196 consensus_queue.cc:260] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.052902 5195 consensus_queue.cc:260] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.053844 5178 heartbeater.cc:503] Master 127.13.223.62:42535 was elected leader, sending a full tablet report...
I20260806 15:56:52.053967 5196 ts_tablet_manager.cc:1428] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 15:56:52.053967 5195 ts_tablet_manager.cc:1428] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 15:56:52.054281 4960 heartbeater.cc:503] Master 127.13.223.62:42535 was elected leader, sending a full tablet report...
W20260806 15:56:52.054458 4961 tablet.cc:2369] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:52.072983 5205 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:52.073199 5205 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.074258 4756 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:52.074424 5205 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737)
I20260806 15:56:52.074496 4756 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.075696 4756 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
I20260806 15:56:52.079499 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.074708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45737 (local address 127.13.223.2:43311)
0806 15:56:52.074842 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.074860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.074951 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.075491 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:52.075494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:52.075513 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:52.075652 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.075660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.076837 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.076840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.078103 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.078113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.079087 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.079091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.079191 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.079214 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:56:52.079302 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 15:56:52.079374 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":40}
I20260806 15:56:52.079499 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.075089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:43311 (local address 127.13.223.1:45737)
0806 15:56:52.075213 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.075217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.075230 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:56:52.075273 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.075275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.075321 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.075437 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:52.075822 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.076707 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.078244 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.078924 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.079089 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.079232 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.079314 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.079377 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 15:56:52.079823 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.074845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:36341 (local address 127.13.223.3:44209)
0806 15:56:52.074969 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.074973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.075424 (+ 451us) server_negotiation.cc:407] Connection header received
0806 15:56:52.075589 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.075604 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.075654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.075729 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:52.076234 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.076953 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.078522 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.079175 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.079263 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.079309 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.079639 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.079712 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:56:52.080044 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357bdf25e1a24c61897f153c85fbaffa" is_pre_election: true
I20260806 15:56:52.080183 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.074707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:44209 (local address 127.13.223.2:36341)
0806 15:56:52.075307 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.075323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.075435 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.075838 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:52.075840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:52.075857 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:52.076055 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.076062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.077094 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.077098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.078395 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.078405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.079399 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.079403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.079498 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.079518 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 15:56:52.079947 (+ 429us) client_negotiation.cc:240] Negotiation successful
0806 15:56:52.080039 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":170,"thread_start_us":144,"threads_started":1}
I20260806 15:56:52.080286 4665 raft_consensus.cc:2454] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178c7a5714dc4493a21b63c26ccbe723 in term 0.
I20260806 15:56:52.080780 4787 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178c7a5714dc4493a21b63c26ccbe723, 357bdf25e1a24c61897f153c85fbaffa; no voters:
I20260806 15:56:52.080945 5103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457ce5cfcabd443f9862ade2e925989a" is_pre_election: true
I20260806 15:56:52.081104 5205 raft_consensus.cc:2781] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:56:52.081200 5205 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:56:52.081254 5205 raft_consensus.cc:3037] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:52.081220 5103 raft_consensus.cc:2454] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178c7a5714dc4493a21b63c26ccbe723 in term 0.
I20260806 15:56:52.081319 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.075809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:44209 (local address 127.13.223.1:36095)
0806 15:56:52.075971 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.075985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.076056 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.077532 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:52.077535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:52.077551 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:52.077651 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.077656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.078752 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.078756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.079934 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.079946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.080880 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.080885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.080998 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.081026 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 15:56:52.081082 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 15:56:52.081174 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":16}
I20260806 15:56:52.081784 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.076893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:36095 (local address 127.13.223.3:44209)
0806 15:56:52.077277 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.077304 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.077318 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:56:52.077359 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.077362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.077401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.077751 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:52.077851 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.078623 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.080072 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.080742 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.081523 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.081566 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.081601 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.081664 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260806 15:56:52.082142 5103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "357bdf25e1a24c61897f153c85fbaffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457ce5cfcabd443f9862ade2e925989a" is_pre_election: true
I20260806 15:56:52.082293 5103 raft_consensus.cc:2454] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357bdf25e1a24c61897f153c85fbaffa in term 0.
I20260806 15:56:52.082643 4557 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357bdf25e1a24c61897f153c85fbaffa, 457ce5cfcabd443f9862ade2e925989a; no voters:
I20260806 15:56:52.082690 5205 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.082924 4756 raft_consensus.cc:2781] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:56:52.083015 4756 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:56:52.083070 4756 raft_consensus.cc:3037] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:52.083316 5205 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 election: Requested vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737)
I20260806 15:56:52.083448 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.077180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:46989 (local address 127.13.223.2:40787)
0806 15:56:52.077278 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.077300 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.077311 (+ 11us) server_negotiation.cc:407] Connection header received
0806 15:56:52.077347 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.077350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.077389 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.077460 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:56:52.077894 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.078572 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.082213 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.082795 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.082911 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.082951 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.083238 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.083314 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:56:52.083462 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.075969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:40787 (local address 127.13.223.1:46989)
0806 15:56:52.076329 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.076339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.076411 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.077622 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:52.077625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:52.077645 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:56:52.077769 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.077775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.078701 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.078704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.082069 (+ 3365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:52.082090 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.082906 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.082909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:52.083026 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.083054 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 15:56:52.083200 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 15:56:52.083283 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":39,"thread_start_us":109,"threads_started":1}
I20260806 15:56:52.084038 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "357bdf25e1a24c61897f153c85fbaffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178c7a5714dc4493a21b63c26ccbe723" is_pre_election: true
I20260806 15:56:52.084264 4889 raft_consensus.cc:2379] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 357bdf25e1a24c61897f153c85fbaffa in current term 1: Already voted for candidate 178c7a5714dc4493a21b63c26ccbe723 in this term.
I20260806 15:56:52.084249 4756 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.084704 4756 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 1 election: Requested vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
I20260806 15:56:52.085201 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "357bdf25e1a24c61897f153c85fbaffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178c7a5714dc4493a21b63c26ccbe723"
I20260806 15:56:52.085350 4889 raft_consensus.cc:2379] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 357bdf25e1a24c61897f153c85fbaffa in current term 1: Already voted for candidate 178c7a5714dc4493a21b63c26ccbe723 in this term.
I20260806 15:56:52.085363 5103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "357bdf25e1a24c61897f153c85fbaffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457ce5cfcabd443f9862ade2e925989a"
I20260806 15:56:52.085484 5103 raft_consensus.cc:3037] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:56:52.085517 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457ce5cfcabd443f9862ade2e925989a"
I20260806 15:56:52.085918 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e321f6fc6841f181b00a9bb1da9488" candidate_uuid: "178c7a5714dc4493a21b63c26ccbe723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357bdf25e1a24c61897f153c85fbaffa"
I20260806 15:56:52.086078 4665 raft_consensus.cc:2379] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 178c7a5714dc4493a21b63c26ccbe723 in current term 1: Already voted for candidate 357bdf25e1a24c61897f153c85fbaffa in this term.
I20260806 15:56:52.086445 4787 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 178c7a5714dc4493a21b63c26ccbe723; no voters: 357bdf25e1a24c61897f153c85fbaffa, 457ce5cfcabd443f9862ade2e925989a
I20260806 15:56:52.086620 5205 raft_consensus.cc:2726] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 15:56:52.086716 5103 raft_consensus.cc:2454] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357bdf25e1a24c61897f153c85fbaffa in term 1.
I20260806 15:56:52.086928 4557 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 357bdf25e1a24c61897f153c85fbaffa, 457ce5cfcabd443f9862ade2e925989a; no voters: 178c7a5714dc4493a21b63c26ccbe723
I20260806 15:56:52.087085 4756 raft_consensus.cc:2781] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:56:52.087322 4756 raft_consensus.cc:686] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 LEADER]: Becoming Leader. State: Replica: 357bdf25e1a24c61897f153c85fbaffa, State: Running, Role: LEADER
I20260806 15:56:52.087574 4756 consensus_queue.cc:237] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.090497 4434 catalog_manager.cc:5495] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1). New cstate: current_term: 1 leader_uuid: "357bdf25e1a24c61897f153c85fbaffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } health_report { overall_health: HEALTHY } } }
W20260806 15:56:52.100262 4748 tablet.cc:2369] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:56:52.109344 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.105777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.13.223.1:45737)
0806 15:56:52.105912 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:52.105917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:52.105931 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:56:52.106073 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:52.106076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:52.106129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:52.106239 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:56:52.106479 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.106979 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.107879 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:52.108101 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:52.108206 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:52.108552 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:56:52.108945 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:56:52.109070 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:52.109144 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 15:56:52.109192 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:56:52.113224 4889 raft_consensus.cc:1261] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Refusing update from remote peer 357bdf25e1a24c61897f153c85fbaffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:52.113797 5211 consensus_queue.cc:1035] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:52.113941 5103 raft_consensus.cc:1261] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Refusing update from remote peer 357bdf25e1a24c61897f153c85fbaffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:56:52.114544 5211 consensus_queue.cc:1035] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:56:52.122951 5219 mvcc.cc:205] Tried to move back new op lower bound from 7315586302409256960 to 7315586302312972288. Current Snapshot: MvccSnapshot[applied={T|T < 7315586302409256960}]
I20260806 15:56:52.130342 4434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:56:52.130510 4434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:56:52.132871 4434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Sending DeleteTablet for 3 replicas of tablet 36e321f6fc6841f181b00a9bb1da9488
I20260806 15:56:52.133721 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e321f6fc6841f181b00a9bb1da9488 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:52 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:36124
I20260806 15:56:52.133950 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e321f6fc6841f181b00a9bb1da9488 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:52 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:44420
I20260806 15:56:52.134279 5237 tablet_replica.cc:330] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: stopping tablet replica
I20260806 15:56:52.134347 4635 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e321f6fc6841f181b00a9bb1da9488 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:56:52 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40366
I20260806 15:56:52.134433 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 4492
I20260806 15:56:52.134527 4762 tablet_replica.cc:330] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: stopping tablet replica
I20260806 15:56:52.134629 5237 raft_consensus.cc:2229] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:56:52.134662 4762 raft_consensus.cc:2229] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:56:52.134950 5237 raft_consensus.cc:2258] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:52.143582 5237 ts_tablet_manager.cc:1905] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:52.147109 5237 ts_tablet_manager.cc:1918] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:52.147274 5237 log.cc:1196] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/36e321f6fc6841f181b00a9bb1da9488
I20260806 15:56:52.148137 5237 ts_tablet_manager.cc:1939] T 36e321f6fc6841f181b00a9bb1da9488 P 457ce5cfcabd443f9862ade2e925989a: Deleting consensus metadata
I20260806 15:56:52.151440 5238 tablet_replica.cc:330] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: stopping tablet replica
I20260806 15:56:52.151957 5238 raft_consensus.cc:2229] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:56:52.152345 5238 raft_consensus.cc:2258] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:56:52.152465 4404 catalog_manager.cc:4841] TS 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209): tablet 36e321f6fc6841f181b00a9bb1da9488 (table test-table [id=90ddf758f6994a48ab89270b47a5c5ff]) successfully deleted
I20260806 15:56:52.153894 5238 ts_tablet_manager.cc:1905] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:56:52.156107 5238 ts_tablet_manager.cc:1918] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:56:52.156194 5238 log.cc:1196] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/36e321f6fc6841f181b00a9bb1da9488
I20260806 15:56:52.156569 5238 ts_tablet_manager.cc:1939] T 36e321f6fc6841f181b00a9bb1da9488 P 178c7a5714dc4493a21b63c26ccbe723: Deleting consensus metadata
I20260806 15:56:52.157801 4407 catalog_manager.cc:4841] TS 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787): tablet 36e321f6fc6841f181b00a9bb1da9488 (table test-table [id=90ddf758f6994a48ab89270b47a5c5ff]) successfully deleted
W20260806 15:56:52.159257 4410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:45737 (error 108)
I20260806 15:56:52.159479 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 4753
I20260806 15:56:52.166335 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.165952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:45737 (local address 127.0.0.1:40398)
0806 15:56:52.166162 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.166234 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:45737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260806 15:56:52.166541 4410 catalog_manager.cc:4584] TS 357bdf25e1a24c61897f153c85fbaffa (127.13.223.1:45737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36e321f6fc6841f181b00a9bb1da9488: Network error: Client connection negotiation failed: client connection to 127.13.223.1:45737: connect: Connection refused (error 111)
I20260806 15:56:52.171921 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 4963
I20260806 15:56:52.183812 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 4380
I20260806 15:56:52.194458 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42535
--webserver_interface=127.13.223.62
--webserver_port=33867
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:52.324899 5241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:52.325201 5241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:52.325286 5241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:52.329274 5241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:56:52.329360 5241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:52.329398 5241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:52.329437 5241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:56:52.329473 5241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:56:52.335323 5241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:42535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:52.336465 5241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:52.337565 5241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:52.346421 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.346449 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.346479 5253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:52.346876 5241 server_base.cc:1034] running on GCE node
I20260806 15:56:52.347520 5241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:52.348760 5241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:52.349954 5241 hybrid_clock.cc:648] HybridClock initialized: now 1786031812349916 us; error 51 us; skew 500 ppm
Aug 06 15:56:52 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:52.352926 5241 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:56:52.353894 5241 webserver.cc:460] Webserver started at http://127.13.223.62:33867/ using document root <none> and password file <none>
I20260806 15:56:52.354478 5241 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:52.354542 5241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:52.358741 5241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 15:56:52.361064 5264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:52.362488 5241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:56:52.362675 5241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "2cedee238cb532e19fc244141fba7360"
server_key_iv: "01e9d2656222dd9649fec8e34bd016a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:52.363240 5241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:52.373474 5241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:52.376252 5241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:52.376475 5241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:52.383677 5241 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:42535
I20260806 15:56:52.383694 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:42535 every 8 connection(s)
I20260806 15:56:52.384840 5241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:56:52.389104 5346 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:56:52.390731 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap starting.
I20260806 15:56:52.391517 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 5241
I20260806 15:56:52.391950 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:45737
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:42535
--webserver_port=45041
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:52.409667 5346 log.cc:824] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:52.422159 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:56:52.422606 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7: Bootstrap complete.
I20260806 15:56:52.427662 5346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:52.428097 5346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b17d2687d4133b7ac2ec0c9c0cec7, State: Initialized, Role: FOLLOWER
I20260806 15:56:52.428707 5346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:52.428838 5346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:56:52.428951 5346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:56:52.429086 5346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 15:56:52.431209 5346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:52.431638 5346 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0b17d2687d4133b7ac2ec0c9c0cec7; no voters:
I20260806 15:56:52.432094 5346 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 15:56:52.432219 5355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 15:56:52.432549 5355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [term 3 LEADER]: Becoming Leader. State: Replica: 3d0b17d2687d4133b7ac2ec0c9c0cec7, State: Running, Role: LEADER
I20260806 15:56:52.432936 5355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } }
I20260806 15:56:52.433590 5346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:56:52.435019 5358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0b17d2687d4133b7ac2ec0c9c0cec7. Latest consensus state: current_term: 3 leader_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } } }
I20260806 15:56:52.435141 5358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:52.435416 5357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b17d2687d4133b7ac2ec0c9c0cec7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 42535 } } }
I20260806 15:56:52.435511 5357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b17d2687d4133b7ac2ec0c9c0cec7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:56:52.435600 5367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:56:52.441646 5367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:56:52.442543 5367 catalog_manager.cc:1259] Loaded cluster ID: 41b43b3db36e4fe58842051b8af0c553
I20260806 15:56:52.442600 5367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 15:56:52.443581 5367 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe0613289ad google::LogMessage::Fail() at ??:0
@ 0x7fe06132b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe0613284de google::LogMessage::Flush() at ??:0
@ 0x7fe061329a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe06bc6bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe06bc6cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe06bc69579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe06bcad640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe06357a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe06357ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe06357c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe06356b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe064baf6db start_thread at ??:0
@ 0x7fe05fe2661f clone at ??:0
W20260806 15:56:52.544413 5350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:52.544874 5350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:52.544973 5350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:52.552139 5350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:52.552242 5350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:52.552376 5350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:56:52.561877 5350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:45737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=45041
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:52.563444 5350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:52.564970 5350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:52.578403 5413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.578413 5412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.578336 5415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:52.578907 5350 server_base.cc:1034] running on GCE node
I20260806 15:56:52.579474 5350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:52.580267 5350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:52.581475 5350 hybrid_clock.cc:648] HybridClock initialized: now 1786031812581443 us; error 40 us; skew 500 ppm
Aug 06 15:56:52 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:52.585143 5350 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:56:52.586212 5350 webserver.cc:460] Webserver started at http://127.13.223.1:45041/ using document root <none> and password file <none>
I20260806 15:56:52.586826 5350 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:52.586934 5350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:52.591455 5350 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:56:52.593983 5425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:52.595528 5350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:56:52.595700 5350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "357bdf25e1a24c61897f153c85fbaffa"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "e9a208a5e6d8aac76cb00b1861ddb9c7"
server_key_iv: "69ebe05b07ae7559eb0b4d095c4ace70"
server_key_version: "encryptionkey@0"
I20260806 15:56:52.596344 5350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:52.606896 5350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:52.610308 5350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:52.610512 5350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:52.611075 5350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:52.612144 5438 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 15:56:52.617493 5350 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 15:56:52.617587 5350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 15:56:52.617712 5350 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 15:56:52.619349 5350 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 15:56:52.619418 5350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 15:56:52.619740 5438 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap starting.
I20260806 15:56:52.630553 5350 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:45737
I20260806 15:56:52.630861 5612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:45737 every 8 connection(s)
I20260806 15:56:52.631781 5350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:56:52.640780 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 5350
I20260806 15:56:52.641286 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:40787
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:42535
--webserver_port=36323
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:56:52.642563 5438 log.cc:824] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Log is configured to *not* fsync() on all Append() calls
I20260806 15:56:52.653254 5438 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:56:52.653772 5438 tablet_bootstrap.cc:492] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Bootstrap complete.
I20260806 15:56:52.655601 5438 ts_tablet_manager.cc:1397] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.007s
I20260806 15:56:52.659464 5438 raft_consensus.cc:348] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.660068 5438 raft_consensus.cc:729] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357bdf25e1a24c61897f153c85fbaffa, State: Initialized, Role: FOLLOWER
I20260806 15:56:52.660686 5438 consensus_queue.cc:260] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:52.661934 5438 ts_tablet_manager.cc:1428] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260806 15:56:52.725773 5615 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:56:52.634211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:58185)
0806 15:56:52.634860 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.634931 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:52.636360 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:52.724949 (+ 88589us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":155,"threads_started":1}
W20260806 15:56:52.726418 5613 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:42535 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.62:42535: Client connection negotiation failed: client connection to 127.13.223.62:42535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 15:56:52.727210 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.726923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:34535)
0806 15:56:52.727050 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.727126 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 15:56:52.727869 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.727647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:43867)
0806 15:56:52.727769 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.727814 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 15:56:52.728027 5613 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 15:56:52.759822 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:52.760110 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:52.760190 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:52.764127 5620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:52.764214 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:52.764303 5620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:56:52.769416 5620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:40787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=36323
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:52.770471 5620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:52.771497 5620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:52.781106 5630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.781159 5629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.781104 5633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:52.781136 5620 server_base.cc:1034] running on GCE node
I20260806 15:56:52.781718 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:52.782333 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:52.783516 5620 hybrid_clock.cc:648] HybridClock initialized: now 1786031812783494 us; error 33 us; skew 500 ppm
Aug 06 15:56:52 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:52.786391 5620 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:56:52.787241 5620 webserver.cc:460] Webserver started at http://127.13.223.2:36323/ using document root <none> and password file <none>
I20260806 15:56:52.787791 5620 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:52.787863 5620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:52.791708 5620 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:56:52.793815 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:52.795046 5620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:56:52.795197 5620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "178c7a5714dc4493a21b63c26ccbe723"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "88f9f8b77ed5d14b4cd83c07e88e8c80"
server_key_iv: "d6af9d1ebf566666f944e45487c08053"
server_key_version: "encryptionkey@0"
I20260806 15:56:52.795732 5620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:52.816778 5620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:52.820204 5620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:52.820436 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:52.821110 5620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:52.821965 5620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:52.822037 5620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:52.822118 5620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:52.822170 5620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:52.831960 5620 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:40787
I20260806 15:56:52.831987 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:40787 every 8 connection(s)
I20260806 15:56:52.833122 5620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:56:52.837306 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.835431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:37759)
0806 15:56:52.835935 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.836064 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
W20260806 15:56:52.837846 5806 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:42535 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.62:42535: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
I20260806 15:56:52.838605 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.838319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:44751)
0806 15:56:52.838466 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.838527 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 15:56:52.839198 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 5620
I20260806 15:56:52.839354 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:52.839143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:56583)
0806 15:56:52.839259 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:52.839302 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 15:56:52.839562 5806 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 15:56:52.839711 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:44209
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:42535
--webserver_port=35611
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:56:52.955875 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:56:52.956130 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:56:52.956171 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:56:52.960210 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:56:52.960270 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:56:52.960359 5809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:56:52.965510 5809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:44209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=35611
--tserver_master_addrs=127.13.223.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:56:52.966622 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:56:52.967634 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:56:52.977643 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.977706 5825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:56:52.977663 5823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:56:52.978327 5809 server_base.cc:1034] running on GCE node
I20260806 15:56:52.978750 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:56:52.979409 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:56:52.980566 5809 hybrid_clock.cc:648] HybridClock initialized: now 1786031812980532 us; error 48 us; skew 500 ppm
Aug 06 15:56:52 dist-test-slave-rwrg krb5kdc[3496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:56:52.983289 5809 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:56:52.984177 5809 webserver.cc:460] Webserver started at http://127.13.223.3:35611/ using document root <none> and password file <none>
I20260806 15:56:52.984812 5809 fs_manager.cc:359] Metadata directory not provided
I20260806 15:56:52.984910 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:56:52.988744 5809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:56:52.990795 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:52.991966 5809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:56:52.992094 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "457ce5cfcabd443f9862ade2e925989a"
format_stamp: "Formatted at 2026-08-06 15:56:50 on dist-test-slave-rwrg"
server_key: "83f7e2ae4d8df0fb1a841f9e9f868304"
server_key_iv: "9c660fd76eff17d319ce3795c70461a4"
server_key_version: "encryptionkey@0"
I20260806 15:56:52.992684 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:56:53.008603 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:56:53.011930 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:56:53.012142 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:56:53.012777 5809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:56:53.013593 5809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:56:53.013671 5809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:53.013741 5809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:56:53.013792 5809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:56:53.023330 5809 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:44209
I20260806 15:56:53.023357 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:44209 every 8 connection(s)
I20260806 15:56:53.024329 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:56:53.027581 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 5809
I20260806 15:56:53.028467 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.026605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:56377)
0806 15:56:53.027098 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.027219 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
W20260806 15:56:53.029021 6008 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:42535 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.62:42535: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
I20260806 15:56:53.029592 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.029385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:43987)
0806 15:56:53.029492 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.029539 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 15:56:53.030298 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.030123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:43033)
0806 15:56:53.030215 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.030256 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 15:56:53.030500 6008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 15:56:53.729147 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.728682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:51499)
0806 15:56:53.728954 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.729016 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260806 15:56:53.803367 6015 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:53.803610 6015 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:53.804821 6015 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
Aug 06 15:56:53 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.3@KRBTEST.COM
Aug 06 15:56:53 dist-test-slave-rwrg krb5kdc[3496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.2@KRBTEST.COM
I20260806 15:56:53.823196 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.805293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:57125 (local address 127.13.223.3:44209)
0806 15:56:53.805606 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:53.805622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:53.805669 (+ 47us) server_negotiation.cc:407] Connection header received
0806 15:56:53.806545 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:53.806601 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:53.807078 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:53.807431 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:53.809481 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.810488 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.811474 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.811911 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.812083 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:53.818254 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:53.818262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:53.818280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:53.818312 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:53.820434 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:53.821111 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:53.821118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:53.821123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:53.821251 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:53.821624 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:53.821627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:53.821629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:53.822025 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:53.822174 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:53.822732 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 15:56:53.822992 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260806 15:56:53.823212 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.805314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:41635 (local address 127.13.223.2:40787)
0806 15:56:53.805729 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 15:56:53.805741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:56:53.805772 (+ 31us) server_negotiation.cc:407] Connection header received
0806 15:56:53.806544 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:56:53.806554 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:56:53.807078 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:56:53.807431 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:56:53.809227 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.810312 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.811464 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.811903 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.812082 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:53.818500 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:56:53.818504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:56:53.818524 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:56:53.818552 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:56:53.820635 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:53.821123 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:53.821128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:53.821132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:53.821248 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:56:53.821623 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:56:53.821626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:56:53.821628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:56:53.822027 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:56:53.822175 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 15:56:53.822729 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 15:56:53.823016 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":106,"threads_started":1}
I20260806 15:56:53.823185 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.805083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:40787 (local address 127.13.223.1:41635)
0806 15:56:53.805524 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.805539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:53.805650 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:53.807526 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:53.807532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:53.808186 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:53.809010 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:53.809030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.810458 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.810462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:53.811341 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:53.811350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.812023 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.812027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:53.812261 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:53.816279 (+ 4018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:53.816296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:53.818314 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:53.820742 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:53.820744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:53.820746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:53.820985 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:53.821354 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:53.821358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:53.821361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:53.821507 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:53.822153 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:53.822160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:53.822426 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 15:56:53.822623 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 15:56:53.822989 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":230,"spinlock_wait_cycles":18560,"thread_start_us":77,"threads_started":1}
I20260806 15:56:53.823300 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.805072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:44209 (local address 127.13.223.1:57125)
0806 15:56:53.805212 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.805228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:56:53.805334 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:56:53.807525 (+ 2191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:56:53.807531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:56:53.808186 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:56:53.809010 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:53.809030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.810637 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.810643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:53.811343 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:56:53.811353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:56:53.812037 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:56:53.812040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:56:53.812260 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:56:53.816299 (+ 4039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:56:53.816311 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:56:53.817945 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:56:53.820608 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:53.820615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:53.820629 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:53.820976 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:53.821388 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:56:53.821390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:56:53.821392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:56:53.821508 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:56:53.822154 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:56:53.822164 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:56:53.822426 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 15:56:53.822623 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 15:56:53.823094 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":393,"spinlock_wait_cycles":28544}
W20260806 15:56:53.826121 5431 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
W20260806 15:56:53.826299 5427 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
I20260806 15:56:53.826364 5431 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 357bdf25e1a24c61897f153c85fbaffa; no voters: 178c7a5714dc4493a21b63c26ccbe723, 457ce5cfcabd443f9862ade2e925989a
I20260806 15:56:53.826633 6015 raft_consensus.cc:2726] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:56:53.840811 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:53.840235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:41363)
0806 15:56:53.840641 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:53.840701 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260806 15:56:54.031481 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:54.031082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:50443)
0806 15:56:54.031320 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:54.031377 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260806 15:56:54.730504 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:54.730066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:41969)
0806 15:56:54.730342 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:54.730403 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260806 15:56:54.841993 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:54.841547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:36901)
0806 15:56:54.841824 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:54.841882 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
I20260806 15:56:55.032819 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:55.032290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:48945)
0806 15:56:55.032642 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:55.032702 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":95,"threads_started":1}
I20260806 15:56:55.709259 6033 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:55.709450 6033 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:55.710026 6033 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
W20260806 15:56:55.710840 5427 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
W20260806 15:56:55.710932 5431 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
I20260806 15:56:55.710979 5427 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 357bdf25e1a24c61897f153c85fbaffa; no voters: 178c7a5714dc4493a21b63c26ccbe723, 457ce5cfcabd443f9862ade2e925989a
I20260806 15:56:55.711194 6033 raft_consensus.cc:2726] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:56:55.731647 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:55.731223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:60633)
0806 15:56:55.731478 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:55.731532 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260806 15:56:55.843279 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:55.842763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:55157)
0806 15:56:55.843134 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:55.843191 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":89,"threads_started":1}
I20260806 15:56:56.034152 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:56.033677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:60935)
0806 15:56:56.033971 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:56.034051 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260806 15:56:56.732944 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:56.732412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:54477)
0806 15:56:56.732787 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:56.732847 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":209,"threads_started":1}
I20260806 15:56:56.844411 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:56.844012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:55633)
0806 15:56:56.844267 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:56.844320 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260806 15:56:57.035573 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:57.035087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:34523)
0806 15:56:57.035371 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:57.035435 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260806 15:56:57.734483 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:57.733964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:43421)
0806 15:56:57.734320 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:57.734384 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20260806 15:56:57.845712 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:57.845226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:45331)
0806 15:56:57.845544 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:57.845611 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
I20260806 15:56:58.036959 6046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:58.036486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:33421)
0806 15:56:58.036798 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:58.036860 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":81,"threads_started":1}
I20260806 15:56:58.735759 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:58.735288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:54093)
0806 15:56:58.735549 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:58.735655 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260806 15:56:58.847102 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:58.846614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:40967)
0806 15:56:58.846931 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:58.846990 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260806 15:56:58.908962 6050 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:56:58.909114 6050 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:56:58.909612 6050 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
W20260806 15:56:58.910416 5431 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
W20260806 15:56:58.910571 5427 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
I20260806 15:56:58.910689 5427 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 357bdf25e1a24c61897f153c85fbaffa; no voters: 178c7a5714dc4493a21b63c26ccbe723, 457ce5cfcabd443f9862ade2e925989a
I20260806 15:56:58.910885 6050 raft_consensus.cc:2726] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:56:59.038187 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:59.037763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:42209)
0806 15:56:59.038035 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:59.038094 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260806 15:56:59.737041 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:59.736595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:50457)
0806 15:56:59.736881 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:59.736940 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260806 15:56:59.848418 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:56:59.847927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:55031)
0806 15:56:59.848263 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 15:56:59.848322 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":97,"threads_started":1}
I20260806 15:57:00.039526 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:00.039011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:44957)
0806 15:57:00.039368 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:00.039430 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":99,"threads_started":1}
I20260806 15:57:00.738425 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:00.737974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:55099)
0806 15:57:00.738265 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:00.738326 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260806 15:57:00.849840 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:00.849351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:48755)
0806 15:57:00.849670 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:00.849733 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260806 15:57:01.040977 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:01.040430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:43389)
0806 15:57:01.040816 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:01.040877 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":198,"threads_started":1}
I20260806 15:57:01.739748 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:01.739271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:57877)
0806 15:57:01.739586 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:01.739646 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260806 15:57:01.851199 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:01.850695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:47423)
0806 15:57:01.851032 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:01.851092 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
I20260806 15:57:02.042363 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:02.041894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:45235)
0806 15:57:02.042203 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:02.042267 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260806 15:57:02.741147 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:02.740661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:42755)
0806 15:57:02.740959 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:02.741032 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260806 15:57:02.852635 6069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:02.852092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:59559)
0806 15:57:02.852367 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:02.852427 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260806 15:57:03.043715 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:03.043250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:42621)
0806 15:57:03.043553 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:03.043615 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20260806 15:57:03.271238 6073 raft_consensus.cc:482] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:03.271369 6073 raft_consensus.cc:504] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457ce5cfcabd443f9862ade2e925989a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 44209 } } peers { permanent_uuid: "178c7a5714dc4493a21b63c26ccbe723" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40787 } } peers { permanent_uuid: "357bdf25e1a24c61897f153c85fbaffa" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 45737 } }
I20260806 15:57:03.271842 6073 leader_election.cc:288] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209), 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787)
W20260806 15:57:03.272600 5427 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 457ce5cfcabd443f9862ade2e925989a (127.13.223.3:44209): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
W20260806 15:57:03.272766 5431 leader_election.cc:341] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 178c7a5714dc4493a21b63c26ccbe723 (127.13.223.2:40787): Not found: Tablet not found: 36e321f6fc6841f181b00a9bb1da9488
I20260806 15:57:03.272841 5431 leader_election.cc:302] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 357bdf25e1a24c61897f153c85fbaffa; no voters: 178c7a5714dc4493a21b63c26ccbe723, 457ce5cfcabd443f9862ade2e925989a
I20260806 15:57:03.272974 6073 raft_consensus.cc:2726] T 36e321f6fc6841f181b00a9bb1da9488 P 357bdf25e1a24c61897f153c85fbaffa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:57:03.742411 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:03.741959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:52853)
0806 15:57:03.742259 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:03.742314 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":128,"threads_started":1}
I20260806 15:57:03.853896 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:03.853408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:41591)
0806 15:57:03.853722 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:03.853798 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260806 15:57:04.045178 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:04.044649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:37825)
0806 15:57:04.044966 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:04.045068 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260806 15:57:04.743965 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:04.743457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:34393)
0806 15:57:04.743797 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:04.743863 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260806 15:57:04.855285 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:04.854795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:33055)
0806 15:57:04.855105 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:04.855180 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20260806 15:57:05.046604 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:05.046111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:50257)
0806 15:57:05.046436 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:05.046498 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260806 15:57:05.745325 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:05.744892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:45447)
0806 15:57:05.745164 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:05.745226 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260806 15:57:05.856709 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:05.856133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:51111)
0806 15:57:05.856447 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:05.856594 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260806 15:57:06.047962 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:06.047459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:57231)
0806 15:57:06.047785 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:06.047846 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260806 15:57:06.746591 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:06.746102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:56607)
0806 15:57:06.746414 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:06.746483 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
I20260806 15:57:06.857896 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:06.857454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:44177)
0806 15:57:06.857745 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:06.857800 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260806 15:57:07.049345 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:07.048898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:51267)
0806 15:57:07.049173 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:07.049229 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
I20260806 15:57:07.747931 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:07.747521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.1:49957)
0806 15:57:07.747779 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:07.747835 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260806 15:57:07.859294 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:07.858846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.2:36537)
0806 15:57:07.859139 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:07.859199 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260806 15:57:08.029750 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 5350
I20260806 15:57:08.041229 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 5620
I20260806 15:57:08.050884 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:08.050376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:42535 (local address 127.13.223.3:43875)
0806 15:57:08.050717 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:08.050781 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":87,"threads_started":1}
I20260806 15:57:08.052986 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 5809
2026-08-06T15:57:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19581 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:57:08 dist-test-slave-rwrg krb5kdc[6093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:57:08 dist-test-slave-rwrg krb5kdc[6093](info): set up 2 sockets
Aug 06 15:57:08 dist-test-slave-rwrg krb5kdc[6093](info): commencing operation
krb5kdc: starting...
W20260806 15:57:09.323962 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:57:09 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:57:09.452121 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44879
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:09.565177 6110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:09.565454 6110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:09.565493 6110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:09.569583 6110 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 15:57:09.569633 6110 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 15:57:09.569648 6110 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 15:57:09.569662 6110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:57:09.569680 6110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:09.569697 6110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:09.569761 6110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:57:09.569782 6110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:57:09.575083 6110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:09.576195 6110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:09.577307 6110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:09.586530 6117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:09.586553 6122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:09.586661 6110 server_base.cc:1034] running on GCE node
W20260806 15:57:09.586531 6118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:09.587417 6110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:09.588585 6110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:09.589797 6110 hybrid_clock.cc:648] HybridClock initialized: now 1786031829589767 us; error 32 us; skew 500 ppm
Aug 06 15:57:09 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031829, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:09.592784 6110 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:57:09.593652 6110 webserver.cc:460] Webserver started at http://127.13.223.62:37483/ using document root <none> and password file <none>
I20260806 15:57:09.594206 6110 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:09.594266 6110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:09.594491 6110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:57:09.596711 6110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "3512c49ae4c84ea8ba597e929d1881e7"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "52536ebc6bc9f468d1685431befd870b"
server_key_iv: "dccc15f6abc55dabfd8e7ee2b39d26e3"
server_key_version: "encryptionkey@0"
I20260806 15:57:09.597285 6110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "3512c49ae4c84ea8ba597e929d1881e7"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "52536ebc6bc9f468d1685431befd870b"
server_key_iv: "dccc15f6abc55dabfd8e7ee2b39d26e3"
server_key_version: "encryptionkey@0"
I20260806 15:57:09.601203 6110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:57:09.603744 6133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:09.605162 6110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:57:09.605307 6110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3512c49ae4c84ea8ba597e929d1881e7"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "52536ebc6bc9f468d1685431befd870b"
server_key_iv: "dccc15f6abc55dabfd8e7ee2b39d26e3"
server_key_version: "encryptionkey@0"
I20260806 15:57:09.605422 6110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:09.625944 6110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:09.629186 6110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:09.629411 6110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:09.636710 6110 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:44879
I20260806 15:57:09.637041 6212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:44879 every 8 connection(s)
I20260806 15:57:09.637766 6110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:57:09.638857 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 6110
I20260806 15:57:09.639002 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:57:09.639299 14204 external_mini_cluster.cc:1428] Setting key 7879449641e3de42fb427e1b94d7ad21
I20260806 15:57:09.641031 6214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:09.647639 6214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap starting.
Aug 06 15:57:09 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:09.649968 6214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:09.650741 6214 log.cc:824] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:09.652745 6214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: No bootstrap required, opened a new log
I20260806 15:57:09.653687 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:09.640725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.13.223.62:44879)
0806 15:57:09.641149 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:09.641170 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:09.641215 (+ 45us) server_negotiation.cc:407] Connection header received
0806 15:57:09.641845 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:09.641860 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:09.642168 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:09.642433 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:09.643389 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:09.644414 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:09.645582 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:09.646045 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:09.646185 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:09.649043 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:09.649048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:09.649061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:09.649088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:09.650848 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:09.651349 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:09.651354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:09.651358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:09.651468 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:09.651759 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:09.651763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:09.651764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:09.652159 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:09.652365 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:09.652719 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 15:57:09.653002 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":145,"threads_started":1}
I20260806 15:57:09.657163 6214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:09.657388 6214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:09.657455 6214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3512c49ae4c84ea8ba597e929d1881e7, State: Initialized, Role: FOLLOWER
I20260806 15:57:09.657929 6214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:09.658083 6214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:57:09.658154 6214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:57:09.658278 6214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:09.659276 6214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:09.659634 6214 leader_election.cc:302] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3512c49ae4c84ea8ba597e929d1881e7; no voters:
I20260806 15:57:09.660008 6214 leader_election.cc:288] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:57:09.660130 6219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:57:09.660427 6219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3512c49ae4c84ea8ba597e929d1881e7, State: Running, Role: LEADER
I20260806 15:57:09.660799 6219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:09.661463 6214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:57:09.662485 6224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3512c49ae4c84ea8ba597e929d1881e7. Latest consensus state: current_term: 1 leader_uuid: "3512c49ae4c84ea8ba597e929d1881e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } } }
I20260806 15:57:09.662616 6224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:57:09.662940 6223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3512c49ae4c84ea8ba597e929d1881e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } } }
I20260806 15:57:09.663050 6223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:57:09.663313 6235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:57:09.666997 6235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:57:09.672943 6235 catalog_manager.cc:1347] Generated new cluster ID: e7f3e8eeb83e4b23a4cb0343d42ad37e
I20260806 15:57:09.673065 6235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:57:09.702893 6235 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:57:09.703912 6235 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:57:09.709790 6235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Generated new TSK 0
I20260806 15:57:09.710675 6235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:57:09.751957 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44879
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:09.864359 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:09.864660 6257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:09.864737 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:09.868698 6257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:09.868746 6257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:09.868849 6257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:57:09.874063 6257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:09.875113 6257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:09.876160 6257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:09.886988 6266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:09.886988 6265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:09.887015 6269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:09.887290 6257 server_base.cc:1034] running on GCE node
I20260806 15:57:09.887689 6257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:09.888338 6257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:09.889519 6257 hybrid_clock.cc:648] HybridClock initialized: now 1786031829889504 us; error 37 us; skew 500 ppm
Aug 06 15:57:09 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031829, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:09.892166 6257 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:57:09.893074 6257 webserver.cc:460] Webserver started at http://127.13.223.1:44333/ using document root <none> and password file <none>
I20260806 15:57:09.893620 6257 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:09.893710 6257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:09.893944 6257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:57:09.896065 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "0850a6fb59f54377a17cb94a872bc6b9"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "66aaac3d2e0178aff97b23c6df014859"
server_key_iv: "649cb6e95eb9183e41b262c51ec9a547"
server_key_version: "encryptionkey@0"
I20260806 15:57:09.896697 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "0850a6fb59f54377a17cb94a872bc6b9"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "66aaac3d2e0178aff97b23c6df014859"
server_key_iv: "649cb6e95eb9183e41b262c51ec9a547"
server_key_version: "encryptionkey@0"
I20260806 15:57:09.900359 6257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:57:09.902863 6282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:09.904031 6257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:57:09.904181 6257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "0850a6fb59f54377a17cb94a872bc6b9"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "66aaac3d2e0178aff97b23c6df014859"
server_key_iv: "649cb6e95eb9183e41b262c51ec9a547"
server_key_version: "encryptionkey@0"
I20260806 15:57:09.904300 6257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:09.929044 6257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:09.931875 6257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:09.932092 6257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:09.932720 6257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:09.933557 6257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:09.933645 6257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:09.933736 6257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:09.933789 6257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:09.943182 6257 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:44383
I20260806 15:57:09.943208 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:44383 every 8 connection(s)
I20260806 15:57:09.944058 6257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:57:09.949154 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 6257
I20260806 15:57:09.949301 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:57:09.949617 14204 external_mini_cluster.cc:1428] Setting key 4c808617042b5285d35109ecf52b6273
Aug 06 15:57:09 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031829, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:09.959194 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:09.945935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:42309 (local address 127.13.223.62:44879)
0806 15:57:09.946081 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:09.946085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:09.947357 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 15:57:09.948262 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:09.948265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:09.948325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:09.948406 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:09.949719 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:09.950414 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:09.952035 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:09.952254 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:09.952335 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:09.955095 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:09.955102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:09.955108 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:09.955140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:09.957002 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:09.957531 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:09.957544 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:09.957548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:09.957647 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:09.957978 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:09.957983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:09.957986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:09.958247 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:09.958354 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:09.958836 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 15:57:09.958976 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 15:57:09.959914 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:09.946308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:42309)
0806 15:57:09.947189 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:09.947233 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:09.948053 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:09.948607 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:09.948614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:09.948960 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:09.949568 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:09.949583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:09.950543 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:09.950546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:09.951901 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:09.951916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:09.952411 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:09.952418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:09.952682 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:09.953437 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:09.953454 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:09.954923 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:09.957155 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:09.957159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:09.957172 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:09.957426 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:09.957750 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:09.957753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:09.957754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:09.957880 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:09.958352 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:09.958357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:09.958573 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 15:57:09.958746 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 15:57:09.958989 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":551,"threads_started":1}
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:57:09.961268 6451 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44879
I20260806 15:57:09.961495 6451 heartbeater.cc:464] Registering TS with master...
I20260806 15:57:09.962133 6451 heartbeater.cc:511] Master 127.13.223.62:44879 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:57:09.963814 6161 ts_manager.cc:194] Registered new tserver with Master: 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383)
I20260806 15:57:09.965494 6161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:42309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:57:09.980554 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44879
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:10.094650 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:10.094954 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:10.095029 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:10.099164 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:10.099255 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:10.099345 6459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:57:10.104619 6459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:10.105638 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:10.106668 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:10.116523 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:10.116576 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:10.116693 6459 server_base.cc:1034] running on GCE node
W20260806 15:57:10.116540 6470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:10.117173 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:10.117856 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:10.119040 6459 hybrid_clock.cc:648] HybridClock initialized: now 1786031830119023 us; error 27 us; skew 500 ppm
Aug 06 15:57:10 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031830, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:10.121944 6459 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:57:10.122834 6459 webserver.cc:460] Webserver started at http://127.13.223.2:36785/ using document root <none> and password file <none>
I20260806 15:57:10.123394 6459 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:10.123486 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:10.123718 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:57:10.125782 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "07f05af2ccbce7ec01f4aacd945ed4d2"
server_key_iv: "50f38795ec6661a34bf0336161db6f1f"
server_key_version: "encryptionkey@0"
I20260806 15:57:10.126365 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "07f05af2ccbce7ec01f4aacd945ed4d2"
server_key_iv: "50f38795ec6661a34bf0336161db6f1f"
server_key_version: "encryptionkey@0"
I20260806 15:57:10.130106 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:57:10.132448 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:10.133723 6459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:57:10.133819 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "07f05af2ccbce7ec01f4aacd945ed4d2"
server_key_iv: "50f38795ec6661a34bf0336161db6f1f"
server_key_version: "encryptionkey@0"
I20260806 15:57:10.133904 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:10.148097 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:10.150971 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:10.151141 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:10.151693 6459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:10.152614 6459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:10.152663 6459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:10.152705 6459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:10.152726 6459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:10.162004 6459 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:38851
I20260806 15:57:10.162057 6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:38851 every 8 connection(s)
I20260806 15:57:10.163178 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:57:10.167273 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 6459
I20260806 15:57:10.167408 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:57:10.167680 14204 external_mini_cluster.cc:1428] Setting key 2dda70d8e696cdc62bde80e7be74fef8
Aug 06 15:57:10 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031830, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:10.178540 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.165239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:43215 (local address 127.13.223.62:44879)
0806 15:57:10.165395 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.165399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.166319 (+ 920us) server_negotiation.cc:407] Connection header received
0806 15:57:10.167801 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.167805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.167862 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.167976 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:10.169451 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.170224 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.171531 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.171765 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.171850 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.174812 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:10.174816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:10.174818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:10.174842 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:10.176389 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:10.176964 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:10.176969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:10.176971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:10.177057 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:10.177435 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:10.177438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:10.177440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:10.177628 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:10.177739 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.178244 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.178369 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:57:10.179455 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.165643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:43215)
0806 15:57:10.166157 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.166204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.167507 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.168181 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.168193 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.168698 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:10.169301 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.169319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.170371 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.170374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.171375 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.171391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.171929 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.171935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.172167 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.172854 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:10.172873 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:10.174637 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:10.176553 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:10.176559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:10.176583 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:10.176854 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:10.177163 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:10.177167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:10.177169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:10.177327 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:10.177740 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:10.177747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:10.177994 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.178182 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.178509 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260806 15:57:10.180708 6672 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44879
I20260806 15:57:10.180949 6672 heartbeater.cc:464] Registering TS with master...
I20260806 15:57:10.181506 6672 heartbeater.cc:511] Master 127.13.223.62:44879 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:57:10.182633 6161 ts_manager.cc:194] Registered new tserver with Master: fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
I20260806 15:57:10.183228 6161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:43215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:57:10.199071 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44879
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:10.315250 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:10.315478 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:10.315521 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:10.319407 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:10.319460 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:10.319515 6683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:57:10.324652 6683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:10.325639 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:10.326576 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:10.336860 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:10.336900 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:10.336843 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:10.337090 6683 server_base.cc:1034] running on GCE node
I20260806 15:57:10.337716 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:10.338440 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:10.339637 6683 hybrid_clock.cc:648] HybridClock initialized: now 1786031830339616 us; error 33 us; skew 500 ppm
Aug 06 15:57:10 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031830, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:10.342600 6683 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:57:10.343549 6683 webserver.cc:460] Webserver started at http://127.13.223.3:36377/ using document root <none> and password file <none>
I20260806 15:57:10.344120 6683 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:10.344182 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:10.344403 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:57:10.346642 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "3fa0d4ea31754ef6b93440b1900d1da1"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "a7a8051c2fcaa10184334ac849a6c544"
server_key_iv: "b50112d4f39d0046d447919bacfd12c7"
server_key_version: "encryptionkey@0"
I20260806 15:57:10.347227 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "3fa0d4ea31754ef6b93440b1900d1da1"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "a7a8051c2fcaa10184334ac849a6c544"
server_key_iv: "b50112d4f39d0046d447919bacfd12c7"
server_key_version: "encryptionkey@0"
I20260806 15:57:10.351176 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 15:57:10.353798 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:10.355119 6683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:57:10.355265 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "3fa0d4ea31754ef6b93440b1900d1da1"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "a7a8051c2fcaa10184334ac849a6c544"
server_key_iv: "b50112d4f39d0046d447919bacfd12c7"
server_key_version: "encryptionkey@0"
I20260806 15:57:10.355386 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:10.384748 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:10.388237 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:10.388463 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:10.389151 6683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:10.389995 6683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:10.390076 6683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:10.390167 6683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:10.390210 6683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:10.399602 6683 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:37433
I20260806 15:57:10.399644 6880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:37433 every 8 connection(s)
I20260806 15:57:10.400619 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:57:10.406378 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 6683
I20260806 15:57:10.406539 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:57:10.406843 14204 external_mini_cluster.cc:1428] Setting key 8d822f3605e08b2bae1960e2638cef6e
Aug 06 15:57:10 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031830, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:10.416103 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.402634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:42049 (local address 127.13.223.62:44879)
0806 15:57:10.402784 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.402788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.403643 (+ 855us) server_negotiation.cc:407] Connection header received
0806 15:57:10.405383 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.405387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.405454 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.405568 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:10.406959 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.407611 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.409267 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.409509 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.409635 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.412290 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:10.412294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:10.412296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:10.412323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:10.414192 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:10.414693 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:10.414696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:10.414697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:10.414778 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:10.415085 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:10.415088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:10.415089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:10.415267 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:10.415357 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.415856 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.415968 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:57:10.417017 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.402933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:42049)
0806 15:57:10.403414 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.403464 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.405128 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.405769 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.405776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.406166 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:10.406797 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.406814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.407761 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.407765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.409132 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.409142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.409671 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.409675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.409863 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.410558 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:10.410574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:10.412123 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:10.414301 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:10.414305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:10.414318 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:10.414571 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:10.414864 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:10.414866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:10.414868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:10.415000 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:10.415365 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:10.415370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:10.415595 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.415791 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.416047 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260806 15:57:10.418354 6881 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44879
I20260806 15:57:10.418672 6881 heartbeater.cc:464] Registering TS with master...
I20260806 15:57:10.419235 6881 heartbeater.cc:511] Master 127.13.223.62:44879 requested a full tablet report, sending...
I20260806 15:57:10.420463 6160 ts_manager.cc:194] Registered new tserver with Master: 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433)
I20260806 15:57:10.421283 6160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:42049
I20260806 15:57:10.421536 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:57:10.432250 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.424085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.13.223.62:44879)
0806 15:57:10.424263 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.424268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.424485 (+ 217us) server_negotiation.cc:407] Connection header received
0806 15:57:10.424651 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.424654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.424692 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.424762 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:10.425599 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.426127 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.427033 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.427247 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.427324 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.428880 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:10.428883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:10.428885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:10.428908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:10.430580 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:10.430997 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:10.431000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:10.431001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:10.431096 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:10.431348 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:10.431351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:10.431352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:10.431618 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:10.431732 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.431914 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.432010 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 15:57:10.435436 6160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:57:10.438735 6160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:57:10.454028 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.447858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:44383 (local address 127.0.0.1:34270)
0806 15:57:10.448816 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.448829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.449001 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.449520 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.449523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.449547 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:10.449653 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.449661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.451022 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.451026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.452116 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.452126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.453309 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.453313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.453526 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.453586 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.453708 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.453862 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":137,"spinlock_wait_cycles":211840,"thread_start_us":89,"threads_started":1}
I20260806 15:57:10.454027 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.447143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38851 (local address 127.0.0.1:60438)
0806 15:57:10.447688 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.447738 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.447871 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.448878 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.448884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.449015 (+ 131us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:10.449278 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.449284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.450504 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.450508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.452174 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.452187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.453309 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.453313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.453530 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.453589 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.453708 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.453874 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":308,"spinlock_wait_cycles":3840,"thread_start_us":79,"threads_started":1}
I20260806 15:57:10.454020 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.448267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.13.223.1:44383)
0806 15:57:10.448665 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.448680 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.449097 (+ 417us) server_negotiation.cc:407] Connection header received
0806 15:57:10.449178 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.449182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.449369 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.449523 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:10.449786 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.450863 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.452251 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.453152 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.453266 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.453408 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.453757 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.453854 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":78,"threads_started":1}
I20260806 15:57:10.455139 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.447265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.13.223.2:38851)
0806 15:57:10.448163 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.448184 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.448207 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:57:10.448286 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.448291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.448517 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.448687 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:10.449477 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.450357 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.452292 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.453148 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.453266 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.453408 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.454882 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.454981 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":154,"threads_started":1}
I20260806 15:57:10.455770 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.447229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37433 (local address 127.0.0.1:53050)
0806 15:57:10.447895 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.447911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.448001 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.448828 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.448836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.448866 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:10.449246 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.449269 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.450743 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.450747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.452199 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.452206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.454564 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.454567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.454695 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.454722 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.455544 (+ 822us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.455633 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":230,"spinlock_wait_cycles":169856,"thread_start_us":122,"threads_started":1}
I20260806 15:57:10.457221 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.447949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.13.223.3:37433)
0806 15:57:10.448304 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.448316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.448341 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:57:10.448420 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.448425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.448636 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.449147 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:10.449380 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.450586 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.453471 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.454230 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.456798 (+ 2568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.456940 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.456992 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.457081 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":142,"threads_started":1}
I20260806 15:57:10.457615 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 69d442e539e4414abb4ca0ca56853236 (DEFAULT_TABLE table=test-table [id=f952fde765e644088069a595c21a01ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:57:10.457636 6352 tablet_service.cc:1467] Processing CreateTablet for tablet 69d442e539e4414abb4ca0ca56853236 (DEFAULT_TABLE table=test-table [id=f952fde765e644088069a595c21a01ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:57:10.458645 6352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d442e539e4414abb4ca0ca56853236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:10.458683 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d442e539e4414abb4ca0ca56853236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:10.459745 6775 tablet_service.cc:1467] Processing CreateTablet for tablet 69d442e539e4414abb4ca0ca56853236 (DEFAULT_TABLE table=test-table [id=f952fde765e644088069a595c21a01ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:57:10.460690 6775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d442e539e4414abb4ca0ca56853236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:10.465543 6907 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap starting.
I20260806 15:57:10.467862 6907 tablet_bootstrap.cc:654] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:10.467952 6909 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Bootstrap starting.
I20260806 15:57:10.468714 6908 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Bootstrap starting.
I20260806 15:57:10.468770 6907 log.cc:824] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:10.470091 6909 tablet_bootstrap.cc:654] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:10.470570 6907 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: No bootstrap required, opened a new log
I20260806 15:57:10.470772 6907 ts_tablet_manager.cc:1397] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 15:57:10.471244 6908 tablet_bootstrap.cc:654] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:10.471477 6909 log.cc:824] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:10.472070 6908 log.cc:824] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:10.473686 6909 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: No bootstrap required, opened a new log
I20260806 15:57:10.473837 6908 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: No bootstrap required, opened a new log
I20260806 15:57:10.473882 6909 ts_tablet_manager.cc:1397] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:57:10.474046 6908 ts_tablet_manager.cc:1397] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 15:57:10.475086 6907 raft_consensus.cc:348] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.475342 6907 raft_consensus.cc:374] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:10.475422 6907 raft_consensus.cc:729] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0850a6fb59f54377a17cb94a872bc6b9, State: Initialized, Role: FOLLOWER
I20260806 15:57:10.475894 6907 consensus_queue.cc:260] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.476601 6451 heartbeater.cc:503] Master 127.13.223.62:44879 was elected leader, sending a full tablet report...
I20260806 15:57:10.476904 6907 ts_tablet_manager.cc:1428] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260806 15:57:10.477288 6908 raft_consensus.cc:348] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.477527 6908 raft_consensus.cc:374] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:10.477591 6908 raft_consensus.cc:729] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf18a36d1f34a6eac2a4b000541ba8a, State: Initialized, Role: FOLLOWER
I20260806 15:57:10.477551 6909 raft_consensus.cc:348] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.477840 6909 raft_consensus.cc:374] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:10.477944 6909 raft_consensus.cc:729] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa0d4ea31754ef6b93440b1900d1da1, State: Initialized, Role: FOLLOWER
I20260806 15:57:10.478155 6908 consensus_queue.cc:260] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.478402 6909 consensus_queue.cc:260] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.478824 6672 heartbeater.cc:503] Master 127.13.223.62:44879 was elected leader, sending a full tablet report...
I20260806 15:57:10.479387 6909 ts_tablet_manager.cc:1428] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260806 15:57:10.479683 6881 heartbeater.cc:503] Master 127.13.223.62:44879 was elected leader, sending a full tablet report...
I20260806 15:57:10.479945 6908 ts_tablet_manager.cc:1428] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:57:10.497746 6921 raft_consensus.cc:482] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:10.497998 6921 raft_consensus.cc:504] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.499179 6921 leader_election.cc:288] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851), 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383)
I20260806 15:57:10.502770 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.499369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38851 (local address 127.13.223.3:56121)
0806 15:57:10.499478 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.499496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.499588 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.499906 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.499909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.499929 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:10.500056 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.500062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.500875 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.500880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.501672 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.501681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.502330 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.502334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.502442 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.502464 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.502565 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.502636 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 15:57:10.502770 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.499505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:56121 (local address 127.13.223.2:38851)
0806 15:57:10.499664 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.499670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.499686 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:57:10.499744 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.499748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.499799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.499889 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:10.500190 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.500763 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.501807 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.502207 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.502338 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.502391 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.502581 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.502661 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:57:10.503227 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.499644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:44383 (local address 127.13.223.3:43157)
0806 15:57:10.500089 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.500100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:10.500216 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:10.500465 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:10.500468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:10.500484 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:10.500640 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.500646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.501355 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.501359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.502139 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:10.502147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.502815 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.502819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:10.502913 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.502934 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:57:10.503027 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 15:57:10.503089 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":171,"threads_started":1}
I20260806 15:57:10.503433 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.499768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:43157 (local address 127.13.223.1:44383)
0806 15:57:10.499923 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.499928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.500179 (+ 251us) server_negotiation.cc:407] Connection header received
0806 15:57:10.500301 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.500304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.500352 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.500453 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:10.500750 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.501239 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.502262 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.502678 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.502815 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.502862 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.503215 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.503299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 15:57:10.503515 6596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d442e539e4414abb4ca0ca56853236" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" is_pre_election: true
I20260806 15:57:10.503883 6596 raft_consensus.cc:2454] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 0.
I20260806 15:57:10.503993 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d442e539e4414abb4ca0ca56853236" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0850a6fb59f54377a17cb94a872bc6b9" is_pre_election: true
I20260806 15:57:10.504283 6383 raft_consensus.cc:2454] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 0.
I20260806 15:57:10.504398 6704 leader_election.cc:302] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a; no voters:
I20260806 15:57:10.504725 6921 raft_consensus.cc:2781] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:57:10.504818 6921 raft_consensus.cc:482] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:57:10.504864 6921 raft_consensus.cc:3037] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:10.505903 6921 raft_consensus.cc:504] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.506243 6921 leader_election.cc:288] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 election: Requested vote from peers fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851), 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383)
I20260806 15:57:10.506578 6596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d442e539e4414abb4ca0ca56853236" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
I20260806 15:57:10.506676 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d442e539e4414abb4ca0ca56853236" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0850a6fb59f54377a17cb94a872bc6b9"
I20260806 15:57:10.506712 6596 raft_consensus.cc:3037] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:10.506839 6383 raft_consensus.cc:3037] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:10.507887 6596 raft_consensus.cc:2454] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 1.
I20260806 15:57:10.508008 6383 raft_consensus.cc:2454] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 1.
I20260806 15:57:10.508214 6704 leader_election.cc:302] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a; no voters:
I20260806 15:57:10.508460 6921 raft_consensus.cc:2781] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:57:10.508755 6921 raft_consensus.cc:686] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa0d4ea31754ef6b93440b1900d1da1, State: Running, Role: LEADER
I20260806 15:57:10.509131 6921 consensus_queue.cc:237] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:10.512547 6160 catalog_manager.cc:5495] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260806 15:57:10.534934 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.531221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.13.223.3:37433)
0806 15:57:10.531397 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:10.531404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:10.531421 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:57:10.531513 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:10.531516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:10.531579 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:10.531702 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:57:10.531957 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.532484 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.533405 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:10.533626 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:10.533739 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:10.534064 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:57:10.534488 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:57:10.534627 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:10.534688 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 15:57:10.534744 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 15:57:10.541009 6596 raft_consensus.cc:1261] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Refusing update from remote peer 3fa0d4ea31754ef6b93440b1900d1da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:57:10.541009 6383 raft_consensus.cc:1261] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Refusing update from remote peer 3fa0d4ea31754ef6b93440b1900d1da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:57:10.541771 6923 consensus_queue.cc:1035] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:57:10.542065 6921 consensus_queue.cc:1035] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:57:10.550925 6936 mvcc.cc:205] Tried to move back new op lower bound from 7315586377888686080 to 7315586377768509440. Current Snapshot: MvccSnapshot[applied={T|T < 7315586377888686080}]
I20260806 15:57:10.558687 6159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:57:10.558898 6159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:57:10.561457 6159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Sending DeleteTablet for 3 replicas of tablet 69d442e539e4414abb4ca0ca56853236
I20260806 15:57:10.562394 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 6257
I20260806 15:57:10.562343 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d442e539e4414abb4ca0ca56853236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:60438
I20260806 15:57:10.562558 6352 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d442e539e4414abb4ca0ca56853236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:34270
I20260806 15:57:10.568653 6775 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d442e539e4414abb4ca0ca56853236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:53050
I20260806 15:57:10.574360 6950 tablet_replica.cc:330] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: stopping tablet replica
I20260806 15:57:10.574354 6952 tablet_replica.cc:330] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: stopping tablet replica
I20260806 15:57:10.574756 6952 raft_consensus.cc:2229] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:57:10.574756 6950 raft_consensus.cc:2229] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:57:10.575134 6950 raft_consensus.cc:2258] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:57:10.575174 6952 raft_consensus.cc:2258] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:57:10.577327 6950 ts_tablet_manager.cc:1905] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:57:10.581182 6950 ts_tablet_manager.cc:1918] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:57:10.581350 6950 log.cc:1196] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/69d442e539e4414abb4ca0ca56853236
I20260806 15:57:10.581764 6950 ts_tablet_manager.cc:1939] T 69d442e539e4414abb4ca0ca56853236 P fcf18a36d1f34a6eac2a4b000541ba8a: Deleting consensus metadata
I20260806 15:57:10.582011 6952 ts_tablet_manager.cc:1905] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:57:10.583240 6138 catalog_manager.cc:4841] TS fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): tablet 69d442e539e4414abb4ca0ca56853236 (table test-table [id=f952fde765e644088069a595c21a01ba]) successfully deleted
I20260806 15:57:10.584826 6952 ts_tablet_manager.cc:1918] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:57:10.584926 6952 log.cc:1196] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/69d442e539e4414abb4ca0ca56853236
I20260806 15:57:10.585672 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 6459
W20260806 15:57:10.585637 6138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:44383 (error 108)
I20260806 15:57:10.585678 6952 ts_tablet_manager.cc:1939] T 69d442e539e4414abb4ca0ca56853236 P 3fa0d4ea31754ef6b93440b1900d1da1: Deleting consensus metadata
I20260806 15:57:10.606549 6141 catalog_manager.cc:4841] TS 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): tablet 69d442e539e4414abb4ca0ca56853236 (table test-table [id=f952fde765e644088069a595c21a01ba]) successfully deleted
I20260806 15:57:10.615581 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:10.615216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:44383 (local address 127.0.0.1:34274)
0806 15:57:10.615386 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:10.615476 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 15:57:10.615888 6138 catalog_manager.cc:4584] TS 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69d442e539e4414abb4ca0ca56853236: Network error: Client connection negotiation failed: client connection to 127.13.223.1:44383: connect: Connection refused (error 111)
I20260806 15:57:10.620548 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 6683
I20260806 15:57:10.629855 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 6110
I20260806 15:57:30.643476 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44879
--webserver_interface=127.13.223.62
--webserver_port=37483
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:30.759878 6956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:30.760135 6956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:30.760174 6956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:30.764031 6956 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 15:57:30.764093 6956 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 15:57:30.764109 6956 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 15:57:30.764122 6956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:57:30.764140 6956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:30.764158 6956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:30.764173 6956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:57:30.764186 6956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:57:30.769541 6956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:30.770598 6956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:30.771730 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:30.786191 6967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:30.786228 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:30.786260 6956 server_base.cc:1034] running on GCE node
W20260806 15:57:30.786348 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:30.787031 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:30.788235 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:30.789441 6956 hybrid_clock.cc:648] HybridClock initialized: now 1786031850789418 us; error 37 us; skew 500 ppm
Aug 06 15:57:30 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031850, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:30.795715 6956 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:57:30.796767 6956 webserver.cc:460] Webserver started at http://127.13.223.62:37483/ using document root <none> and password file <none>
I20260806 15:57:30.797413 6956 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:30.797526 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:30.801591 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:57:30.803817 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:30.805163 6956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:57:30.805335 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3512c49ae4c84ea8ba597e929d1881e7"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "52536ebc6bc9f468d1685431befd870b"
server_key_iv: "dccc15f6abc55dabfd8e7ee2b39d26e3"
server_key_version: "encryptionkey@0"
I20260806 15:57:30.805982 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:30.823303 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:30.826721 6956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:30.826936 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:30.834841 6956 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:44879
I20260806 15:57:30.835189 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:44879 every 8 connection(s)
I20260806 15:57:30.836027 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:57:30.840651 7061 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:57:30.841667 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 6956
I20260806 15:57:30.842427 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:44383
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:44879
--webserver_port=44333
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:57:30.842980 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap starting.
I20260806 15:57:30.868680 7061 log.cc:824] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:30.881085 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:57:30.881623 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap complete.
I20260806 15:57:30.887537 7061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:30.888125 7061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3512c49ae4c84ea8ba597e929d1881e7, State: Initialized, Role: FOLLOWER
I20260806 15:57:30.888792 7061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:30.888968 7061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:57:30.889048 7061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:57:30.889156 7061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 15:57:30.891945 7061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:30.892340 7061 leader_election.cc:302] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3512c49ae4c84ea8ba597e929d1881e7; no voters:
I20260806 15:57:30.892768 7061 leader_election.cc:288] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 15:57:30.892921 7065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 15:57:30.893256 7065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 LEADER]: Becoming Leader. State: Replica: 3512c49ae4c84ea8ba597e929d1881e7, State: Running, Role: LEADER
I20260806 15:57:30.893644 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:30.894165 7061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:57:30.895388 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3512c49ae4c84ea8ba597e929d1881e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } } }
I20260806 15:57:30.895443 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3512c49ae4c84ea8ba597e929d1881e7. Latest consensus state: current_term: 2 leader_uuid: "3512c49ae4c84ea8ba597e929d1881e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } } }
I20260806 15:57:30.895526 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:57:30.895526 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:57:30.896299 7079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:57:30.900888 7079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:57:30.901476 7079 catalog_manager.cc:1259] Loaded cluster ID: e7f3e8eeb83e4b23a4cb0343d42ad37e
I20260806 15:57:30.901562 7079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:57:30.903725 7079 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:57:30.904364 7079 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Loaded TSK: 0
I20260806 15:57:30.921547 7079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Generated new TSK 1
I20260806 15:57:30.921903 7079 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 15:57:30.990163 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:30.990458 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:30.990541 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:30.994786 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:30.994876 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:30.994961 7063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:57:31.000391 7063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:44383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=44333
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:31.001529 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:31.002578 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:31.012341 7114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:31.012349 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:31.012632 7117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:31.013477 7063 server_base.cc:1034] running on GCE node
I20260806 15:57:31.013955 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:31.014734 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:31.015941 7063 hybrid_clock.cc:648] HybridClock initialized: now 1786031851015917 us; error 32 us; skew 500 ppm
Aug 06 15:57:31 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031851, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:31.019343 7063 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:57:31.020335 7063 webserver.cc:460] Webserver started at http://127.13.223.1:44333/ using document root <none> and password file <none>
I20260806 15:57:31.020959 7063 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:31.021037 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:31.025279 7063 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:57:31.027813 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.029320 7063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:57:31.029474 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "0850a6fb59f54377a17cb94a872bc6b9"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "66aaac3d2e0178aff97b23c6df014859"
server_key_iv: "649cb6e95eb9183e41b262c51ec9a547"
server_key_version: "encryptionkey@0"
I20260806 15:57:31.030100 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:31.041759 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:31.045264 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:31.045486 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:31.046123 7063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:31.047322 7137 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 15:57:31.051807 7063 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 15:57:31.051900 7063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 15:57:31.051985 7063 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 15:57:31.053790 7063 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 15:57:31.053860 7063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 15:57:31.054117 7137 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap starting.
I20260806 15:57:31.065303 7063 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:44383
I20260806 15:57:31.065433 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:44383 every 8 connection(s)
I20260806 15:57:31.066565 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:57:31.072484 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 7063
I20260806 15:57:31.073072 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:38851
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:44879
--webserver_port=36785
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:57:31.076318 7137 log.cc:824] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Log is configured to *not* fsync() on all Append() calls
Aug 06 15:57:31 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031851, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:31.085510 7137 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:57:31.086066 7137 tablet_bootstrap.cc:492] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap complete.
I20260806 15:57:31.086624 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.068891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:60909 (local address 127.13.223.62:44879)
0806 15:57:31.069322 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.069346 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.069694 (+ 348us) server_negotiation.cc:407] Connection header received
0806 15:57:31.070870 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.070882 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.071187 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.071567 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:31.072925 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.074068 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.075545 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.076255 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.076453 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.079898 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:31.079905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:31.079922 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:31.079959 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:31.082320 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.083106 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.083112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.083117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.083272 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.083776 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.083780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.083781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.084245 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:31.084381 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.085250 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.085585 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":151,"threads_started":1}
I20260806 15:57:31.086946 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.068901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:60909)
0806 15:57:31.069502 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.069547 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.070416 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.071606 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.071615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.072080 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:31.072722 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.072749 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.074255 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.074259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.075382 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.075398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.076433 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.076466 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.076686 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.077464 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:31.077484 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:31.079637 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:31.082475 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:31.082483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:31.082510 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:31.082939 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:31.083398 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:31.083401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:31.083405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:31.083642 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:31.084397 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:31.084407 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:31.084820 (+ 413us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.085079 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.085493 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":178,"threads_started":1}
I20260806 15:57:31.088039 7137 ts_tablet_manager.cc:1397] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.004s
I20260806 15:57:31.089571 7307 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44879
I20260806 15:57:31.089870 7307 heartbeater.cc:464] Registering TS with master...
I20260806 15:57:31.090456 7307 heartbeater.cc:511] Master 127.13.223.62:44879 requested a full tablet report, sending...
I20260806 15:57:31.091060 7137 raft_consensus.cc:348] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.091634 7137 raft_consensus.cc:729] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0850a6fb59f54377a17cb94a872bc6b9, State: Initialized, Role: FOLLOWER
I20260806 15:57:31.092182 7137 consensus_queue.cc:260] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.093268 7137 ts_tablet_manager.cc:1428] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 15:57:31.093290 7005 ts_manager.cc:194] Registered new tserver with Master: 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383)
I20260806 15:57:31.097052 7005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:60909
I20260806 15:57:31.099445 7307 heartbeater.cc:503] Master 127.13.223.62:44879 was elected leader, sending a full tablet report...
I20260806 15:57:31.103755 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.096272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:44383 (local address 127.0.0.1:37516)
0806 15:57:31.097163 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.097207 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.097329 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.099923 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.099931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.099961 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:31.100100 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.100116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.101243 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.101247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.102149 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.102159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.103252 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.103259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.103390 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.103431 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.103541 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.103644 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":134,"threads_started":1}
I20260806 15:57:31.103830 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.096386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.13.223.1:44383)
0806 15:57:31.099413 (+ 3027us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.099425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.099451 (+ 26us) server_negotiation.cc:407] Connection header received
0806 15:57:31.099527 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.099533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.099750 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.099919 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:31.100257 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.101081 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.102291 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.103099 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.103219 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.103353 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.103605 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.103700 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2916,"thread_start_us":194,"threads_started":1}
I20260806 15:57:31.104856 7201 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d442e539e4414abb4ca0ca56853236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:37516
I20260806 15:57:31.105690 7324 tablet_replica.cc:330] stopping tablet replica
I20260806 15:57:31.105984 7324 raft_consensus.cc:2229] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:57:31.106232 7324 raft_consensus.cc:2258] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:57:31.107218 7324 ts_tablet_manager.cc:1905] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:57:31.110968 7324 ts_tablet_manager.cc:1918] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:57:31.111050 7324 log.cc:1196] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal/wals/69d442e539e4414abb4ca0ca56853236
I20260806 15:57:31.111330 7324 ts_tablet_manager.cc:1939] T 69d442e539e4414abb4ca0ca56853236 P 0850a6fb59f54377a17cb94a872bc6b9: Deleting consensus metadata
I20260806 15:57:31.113291 6982 catalog_manager.cc:4841] TS 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383): tablet 69d442e539e4414abb4ca0ca56853236 (table test-table [id=f952fde765e644088069a595c21a01ba]) successfully deleted
W20260806 15:57:31.196295 7316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:31.196606 7316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:31.196713 7316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:31.200726 7316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:31.200805 7316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:31.200891 7316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:57:31.206044 7316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:38851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=36785
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:31.207172 7316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:31.208246 7316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:57:31.218389 7316 server_base.cc:1034] running on GCE node
W20260806 15:57:31.218353 7336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:31.218354 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:31.218472 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:31.219095 7316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:31.219754 7316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:31.220963 7316 hybrid_clock.cc:648] HybridClock initialized: now 1786031851220948 us; error 31 us; skew 500 ppm
Aug 06 15:57:31 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031851, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:31.224233 7316 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:57:31.225226 7316 webserver.cc:460] Webserver started at http://127.13.223.2:36785/ using document root <none> and password file <none>
I20260806 15:57:31.225795 7316 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:31.225909 7316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:31.229902 7316 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:57:31.232214 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.233661 7316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:57:31.233808 7316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "07f05af2ccbce7ec01f4aacd945ed4d2"
server_key_iv: "50f38795ec6661a34bf0336161db6f1f"
server_key_version: "encryptionkey@0"
I20260806 15:57:31.234473 7316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:31.267757 7316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:31.270948 7316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:31.271183 7316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:31.271852 7316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:31.272807 7316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:31.272889 7316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.272966 7316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:31.273005 7316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.282673 7316 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:38851
I20260806 15:57:31.282699 7547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:38851 every 8 connection(s)
I20260806 15:57:31.283818 7316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:57:31.292711 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 7316
I20260806 15:57:31.293188 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:37433
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:44879
--webserver_port=36377
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:57:31 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031851, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:31.298756 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.285870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:52101 (local address 127.13.223.62:44879)
0806 15:57:31.286016 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.286020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.286874 (+ 854us) server_negotiation.cc:407] Connection header received
0806 15:57:31.288266 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.288271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.288333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.288439 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:31.289656 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.290183 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.291330 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.291580 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.291707 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.294403 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:31.294407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:31.294409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:31.294433 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:31.296304 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.296906 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.296912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.296917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.297013 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.297379 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.297385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.297389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.297661 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:31.297768 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.298362 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.298528 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:57:31.299472 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.286237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:52101)
0806 15:57:31.286706 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.286749 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.288031 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.288644 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.288651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.289000 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:31.289490 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.289504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.290316 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.290320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.291194 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.291203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.291750 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.291755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.291924 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.292610 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:31.292627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:31.294221 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:31.296455 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:31.296461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:31.296474 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:31.296809 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:31.297115 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:31.297117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:31.297119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:31.297264 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:31.297799 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:31.297804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:31.298062 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.298237 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.298494 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20260806 15:57:31.300900 7548 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44879
I20260806 15:57:31.301182 7548 heartbeater.cc:464] Registering TS with master...
I20260806 15:57:31.301836 7548 heartbeater.cc:511] Master 127.13.223.62:44879 requested a full tablet report, sending...
I20260806 15:57:31.303097 7005 ts_manager.cc:194] Registered new tserver with Master: fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
I20260806 15:57:31.303865 7005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:52101
W20260806 15:57:31.417069 7556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:31.417341 7556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:31.417439 7556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:31.421975 7556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:31.422050 7556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:31.422173 7556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:57:31.427709 7556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:37433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=36377
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:31.428834 7556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:31.429934 7556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:31.440178 7569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:31.440176 7562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:31.440261 7564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:31.440542 7556 server_base.cc:1034] running on GCE node
I20260806 15:57:31.441171 7556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:31.441846 7556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:31.443048 7556 hybrid_clock.cc:648] HybridClock initialized: now 1786031851443011 us; error 48 us; skew 500 ppm
Aug 06 15:57:31 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031851, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:31.445834 7556 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:57:31.446733 7556 webserver.cc:460] Webserver started at http://127.13.223.3:36377/ using document root <none> and password file <none>
I20260806 15:57:31.447305 7556 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:31.447428 7556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:31.451622 7556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:57:31.453863 7582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.455042 7556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:57:31.455194 7556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "3fa0d4ea31754ef6b93440b1900d1da1"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "a7a8051c2fcaa10184334ac849a6c544"
server_key_iv: "b50112d4f39d0046d447919bacfd12c7"
server_key_version: "encryptionkey@0"
I20260806 15:57:31.455739 7556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:31.499496 7556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:31.502717 7556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:31.502938 7556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:31.503582 7556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:31.504462 7556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:31.504561 7556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.504637 7556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:31.504684 7556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:31.514964 7556 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:37433
I20260806 15:57:31.514986 7788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:37433 every 8 connection(s)
I20260806 15:57:31.515990 7556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:57:31.522421 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 7556
Aug 06 15:57:31 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031851, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:57:31.530537 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.517981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:37601 (local address 127.13.223.62:44879)
0806 15:57:31.518146 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.518151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.519047 (+ 896us) server_negotiation.cc:407] Connection header received
0806 15:57:31.520443 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.520446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.520529 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.520645 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:31.521845 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.522387 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.523877 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.524106 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.524217 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.526855 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:31.526859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:31.526862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:31.526892 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:31.528418 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.528967 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.528971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.528973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.529055 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.529462 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.529467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.529469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.529645 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:31.529760 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.530245 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.530357 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 15:57:31.531339 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.518404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:37601)
0806 15:57:31.518887 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.518931 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.520225 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.520829 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.520855 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.521197 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:31.521692 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.521713 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.522531 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.522536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.523740 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.523750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.524267 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.524271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.524434 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.525181 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:31.525204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:31.526689 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:31.528563 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:31.528569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:31.528583 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:31.528854 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:31.529180 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:31.529184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:31.529186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:31.529355 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:31.529759 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:31.529766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:31.530000 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.530200 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.530510 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260806 15:57:31.532332 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.523246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.13.223.62:44879)
0806 15:57:31.523522 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.523577 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.523733 (+ 156us) server_negotiation.cc:407] Connection header received
0806 15:57:31.523914 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.523918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.523965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.524068 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:31.525012 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.525578 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.526609 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.526855 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.526978 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.528409 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:31.528413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:31.528416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:31.528447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:31.530148 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.530777 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.530784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.530786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.530883 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.531196 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.531201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.531203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.531394 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:31.531638 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.532024 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.532142 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":125,"threads_started":1}
I20260806 15:57:31.532747 7790 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44879
I20260806 15:57:31.532996 7790 heartbeater.cc:464] Registering TS with master...
I20260806 15:57:31.533640 7790 heartbeater.cc:511] Master 127.13.223.62:44879 requested a full tablet report, sending...
I20260806 15:57:31.534739 7005 ts_manager.cc:194] Registered new tserver with Master: 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433)
I20260806 15:57:31.535293 7005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:37601
I20260806 15:57:31.546324 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:57:31.550845 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.546978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.13.223.62:44879)
0806 15:57:31.547128 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.547131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.547413 (+ 282us) server_negotiation.cc:407] Connection header received
0806 15:57:31.547505 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.547508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.547626 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.547726 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:57:31.547955 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.548591 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.549575 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.549928 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.550027 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.550328 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:57:31.550466 (+ 138us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 15:57:31.550688 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58884: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 15:57:31.551069 7793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58884: token expired
W20260806 15:57:31.551332 6885 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.223.62:44879)
I20260806 15:57:31.558954 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.551735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.13.223.62:44879)
0806 15:57:31.551874 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.551881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.551895 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:57:31.551971 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.551973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.552010 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.552076 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:31.552879 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.553447 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.554276 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.554489 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.554568 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.556035 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:31.556038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:31.556040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:31.556060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:31.557423 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.557849 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.557851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.557853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.557925 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:31.558163 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:31.558165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:31.558166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:31.558356 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:31.558455 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.558685 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.558774 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:57:31.561477 7005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:57:31.564734 7005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:57:31.572978 7201 tablet_service.cc:1467] Processing CreateTablet for tablet d65f451c1907442fbcffbc74d9a01167 (DEFAULT_TABLE table=test-table [id=29fa61199794472288efb27c7926f176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:57:31.573668 7201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65f451c1907442fbcffbc74d9a01167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:31.576234 7137 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap starting.
I20260806 15:57:31.577327 7137 tablet_bootstrap.cc:654] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:31.577800 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.572210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.13.223.2:38851)
0806 15:57:31.572673 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.572684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.572706 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:57:31.572779 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.572784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.572968 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.573120 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:31.573373 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.574365 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.575932 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.576850 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.576955 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.577120 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.577348 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.577451 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":145,"threads_started":1}
I20260806 15:57:31.578059 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.571674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37433 (local address 127.0.0.1:41198)
0806 15:57:31.571819 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.571836 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.571924 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.572903 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.572906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.572925 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:31.573027 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.573042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.574432 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.574435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.575784 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.575796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.577620 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.577624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.577716 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.577743 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.577796 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.577877 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":137}
I20260806 15:57:31.577910 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.571985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38851 (local address 127.0.0.1:38164)
0806 15:57:31.572428 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.572442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.572570 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.573120 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.573122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.573140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:31.573240 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.573248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.574494 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.574497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.575787 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.575796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.576978 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.576981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.577086 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.577113 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.577196 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.577695 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":47,"thread_start_us":93,"threads_started":1}
I20260806 15:57:31.578557 7137 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: No bootstrap required, opened a new log
I20260806 15:57:31.578569 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.571856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.13.223.3:37433)
0806 15:57:31.572352 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.572368 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.572395 (+ 27us) server_negotiation.cc:407] Connection header received
0806 15:57:31.572475 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.572480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.572743 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.572909 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:31.573179 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.574300 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.576643 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.577470 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.577984 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.578251 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.578315 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.578415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":219,"threads_started":1}
I20260806 15:57:31.578680 7137 ts_tablet_manager.cc:1397] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 15:57:31.579368 7137 raft_consensus.cc:348] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.579486 7137 raft_consensus.cc:374] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:31.579526 7137 raft_consensus.cc:729] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0850a6fb59f54377a17cb94a872bc6b9, State: Initialized, Role: FOLLOWER
I20260806 15:57:31.579643 7137 consensus_queue.cc:260] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.579882 7137 ts_tablet_manager.cc:1428] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 15:57:31.581467 7671 tablet_service.cc:1467] Processing CreateTablet for tablet d65f451c1907442fbcffbc74d9a01167 (DEFAULT_TABLE table=test-table [id=29fa61199794472288efb27c7926f176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:57:31.581450 7432 tablet_service.cc:1467] Processing CreateTablet for tablet d65f451c1907442fbcffbc74d9a01167 (DEFAULT_TABLE table=test-table [id=29fa61199794472288efb27c7926f176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:57:31.582489 7432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65f451c1907442fbcffbc74d9a01167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:31.582548 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65f451c1907442fbcffbc74d9a01167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:57:31.588864 7808 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Bootstrap starting.
I20260806 15:57:31.588922 7809 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Bootstrap starting.
I20260806 15:57:31.590929 7809 tablet_bootstrap.cc:654] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:31.591259 7808 tablet_bootstrap.cc:654] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Neither blocks nor log segments found. Creating new log.
I20260806 15:57:31.592186 7808 log.cc:824] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:31.592216 7809 log.cc:824] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:31.593902 7808 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: No bootstrap required, opened a new log
I20260806 15:57:31.593914 7809 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: No bootstrap required, opened a new log
I20260806 15:57:31.594110 7808 ts_tablet_manager.cc:1397] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 15:57:31.594112 7809 ts_tablet_manager.cc:1397] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 15:57:31.597385 7808 raft_consensus.cc:348] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.597388 7809 raft_consensus.cc:348] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.597633 7808 raft_consensus.cc:374] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:31.597625 7809 raft_consensus.cc:374] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:57:31.597693 7809 raft_consensus.cc:729] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa0d4ea31754ef6b93440b1900d1da1, State: Initialized, Role: FOLLOWER
I20260806 15:57:31.597692 7808 raft_consensus.cc:729] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf18a36d1f34a6eac2a4b000541ba8a, State: Initialized, Role: FOLLOWER
I20260806 15:57:31.598116 7809 consensus_queue.cc:260] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.598120 7808 consensus_queue.cc:260] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.598737 7790 heartbeater.cc:503] Master 127.13.223.62:44879 was elected leader, sending a full tablet report...
I20260806 15:57:31.598794 7548 heartbeater.cc:503] Master 127.13.223.62:44879 was elected leader, sending a full tablet report...
I20260806 15:57:31.599138 7808 ts_tablet_manager.cc:1428] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260806 15:57:31.600045 7809 ts_tablet_manager.cc:1428] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 15:57:31.767062 7791 tablet.cc:2369] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:57:31.785090 7549 tablet.cc:2369] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:57:31.789917 7814 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:31.790163 7814 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.791381 7814 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851), 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383)
I20260806 15:57:31.797104 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.792129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:33037 (local address 127.13.223.1:44383)
0806 15:57:31.792373 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.792378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.792393 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:57:31.792515 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.792522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.792570 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.792665 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:31.792924 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.793728 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.795712 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.796369 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.796542 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.796631 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.796886 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.796965 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260806 15:57:31.797120 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.791759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:56493 (local address 127.13.223.2:38851)
0806 15:57:31.791912 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.791916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.791932 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:57:31.792357 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.792361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.792422 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.792534 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:57:31.792897 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.793700 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.795712 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.796363 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.796542 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.796598 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.796886 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.796965 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 15:57:31.797135 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.791652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38851 (local address 127.13.223.3:56493)
0806 15:57:31.791806 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.791827 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.791946 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.792585 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.792587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.792602 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:31.792749 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.792758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.793849 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.793865 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.795570 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.795582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.796540 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.796545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.796707 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.796776 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.796855 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.796969 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":285}
I20260806 15:57:31.797144 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.791997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:44383 (local address 127.13.223.3:33037)
0806 15:57:31.792314 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.792326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:31.792412 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:31.792680 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:31.792682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:31.792693 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:57:31.792805 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.792814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.793849 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.793865 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.795571 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:31.795582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.796540 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.796545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:31.796701 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.796772 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 15:57:31.796855 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 15:57:31.796969 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":263,"thread_start_us":149,"threads_started":1}
I20260806 15:57:31.797904 7468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f451c1907442fbcffbc74d9a01167" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" is_pre_election: true
I20260806 15:57:31.797960 7233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f451c1907442fbcffbc74d9a01167" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0850a6fb59f54377a17cb94a872bc6b9" is_pre_election: true
I20260806 15:57:31.798264 7233 raft_consensus.cc:2454] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 0.
I20260806 15:57:31.798274 7468 raft_consensus.cc:2454] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 0.
I20260806 15:57:31.798756 7585 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0850a6fb59f54377a17cb94a872bc6b9, 3fa0d4ea31754ef6b93440b1900d1da1; no voters:
I20260806 15:57:31.799039 7814 raft_consensus.cc:2781] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:57:31.799131 7814 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:57:31.799165 7814 raft_consensus.cc:3037] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:31.800247 7814 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.800657 7814 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 election: Requested vote from peers fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851), 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383)
I20260806 15:57:31.801002 7468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f451c1907442fbcffbc74d9a01167" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
I20260806 15:57:31.801108 7468 raft_consensus.cc:3037] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:31.801132 7233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f451c1907442fbcffbc74d9a01167" candidate_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0850a6fb59f54377a17cb94a872bc6b9"
I20260806 15:57:31.801283 7233 raft_consensus.cc:3037] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:57:31.802217 7468 raft_consensus.cc:2454] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 1.
I20260806 15:57:31.802417 7233 raft_consensus.cc:2454] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa0d4ea31754ef6b93440b1900d1da1 in term 1.
I20260806 15:57:31.802577 7585 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a; no voters:
I20260806 15:57:31.802758 7814 raft_consensus.cc:2781] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:57:31.802991 7814 raft_consensus.cc:686] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa0d4ea31754ef6b93440b1900d1da1, State: Running, Role: LEADER
I20260806 15:57:31.803349 7814 consensus_queue.cc:237] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:31.806716 7005 catalog_manager.cc:5495] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } health_report { overall_health: UNKNOWN } } }
W20260806 15:57:31.818074 7309 tablet.cc:2369] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:57:31.819967 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.816352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.13.223.3:37433)
0806 15:57:31.816472 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:31.816475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:31.816545 (+ 70us) server_negotiation.cc:407] Connection header received
0806 15:57:31.816632 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:31.816636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:31.816699 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:31.816836 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:57:31.817104 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.817606 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.818487 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:31.818705 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:31.818810 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:31.819126 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:57:31.819518 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:57:31.819694 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:31.819757 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 15:57:31.819812 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:57:31.825935 7468 raft_consensus.cc:1261] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Refusing update from remote peer 3fa0d4ea31754ef6b93440b1900d1da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:57:31.825932 7233 raft_consensus.cc:1261] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Refusing update from remote peer 3fa0d4ea31754ef6b93440b1900d1da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:57:31.826689 7814 consensus_queue.cc:1035] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:57:31.826934 7820 consensus_queue.cc:1035] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:57:31.834378 7824 mvcc.cc:205] Tried to move back new op lower bound from 7315586465071312896 to 7315586464989151232. Current Snapshot: MvccSnapshot[applied={T|T < 7315586465071312896}]
I20260806 15:57:31.843376 7003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:57:31.843545 7003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:57:31.845883 7003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Sending DeleteTablet for 3 replicas of tablet d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:31.846596 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet d65f451c1907442fbcffbc74d9a01167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:31 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:41198
I20260806 15:57:31.846751 7201 tablet_service.cc:1514] Processing DeleteTablet for tablet d65f451c1907442fbcffbc74d9a01167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:31 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:37516
I20260806 15:57:31.846808 7432 tablet_service.cc:1514] Processing DeleteTablet for tablet d65f451c1907442fbcffbc74d9a01167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:57:31 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:38164
I20260806 15:57:31.847087 7837 tablet_replica.cc:330] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: stopping tablet replica
I20260806 15:57:31.847194 7825 mvcc.cc:205] Tried to move back new op lower bound from 7315586465071312896 to 7315586464989151232. Current Snapshot: MvccSnapshot[applied={T|T < 7315586465071312896 or (T in {7315586465071312896})}]
I20260806 15:57:31.847298 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 7063
I20260806 15:57:31.847442 7837 raft_consensus.cc:2229] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:57:31.859090 7837 raft_consensus.cc:2258] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:57:31.861843 7841 tablet_replica.cc:330] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: stopping tablet replica
I20260806 15:57:31.861881 7837 ts_tablet_manager.cc:1905] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:57:31.862191 7841 raft_consensus.cc:2229] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:57:31.862540 7841 raft_consensus.cc:2258] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:57:31.864073 7841 ts_tablet_manager.cc:1905] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:57:31.864953 7837 ts_tablet_manager.cc:1918] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:57:31.865067 7837 log.cc:1196] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:31.865465 7837 ts_tablet_manager.cc:1939] T d65f451c1907442fbcffbc74d9a01167 P 3fa0d4ea31754ef6b93440b1900d1da1: Deleting consensus metadata
I20260806 15:57:31.866829 6983 catalog_manager.cc:4841] TS 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): tablet d65f451c1907442fbcffbc74d9a01167 (table test-table [id=29fa61199794472288efb27c7926f176]) successfully deleted
I20260806 15:57:31.866956 7841 ts_tablet_manager.cc:1918] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:57:31.867095 7841 log.cc:1196] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:31.867431 7841 ts_tablet_manager.cc:1939] T d65f451c1907442fbcffbc74d9a01167 P fcf18a36d1f34a6eac2a4b000541ba8a: Deleting consensus metadata
I20260806 15:57:31.868475 6982 catalog_manager.cc:4841] TS fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): tablet d65f451c1907442fbcffbc74d9a01167 (table test-table [id=29fa61199794472288efb27c7926f176]) successfully deleted
W20260806 15:57:31.870023 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:44383 (error 108)
I20260806 15:57:31.871081 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:31.870774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:44383 (local address 127.0.0.1:37526)
0806 15:57:31.870924 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:31.870999 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 15:57:31.871260 6982 catalog_manager.cc:4584] TS 0850a6fb59f54377a17cb94a872bc6b9 (127.13.223.1:44383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d65f451c1907442fbcffbc74d9a01167: Network error: Client connection negotiation failed: client connection to 127.13.223.1:44383: connect: Connection refused (error 111)
I20260806 15:57:31.872725 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 7316
I20260806 15:57:31.887009 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 7556
I20260806 15:57:31.896136 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 6956
I20260806 15:57:51.910841 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44879
--webserver_interface=127.13.223.62
--webserver_port=37483
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:52.025435 7844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:52.025734 7844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:52.025804 7844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:52.029848 7844 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 15:57:52.029928 7844 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 15:57:52.029955 7844 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 15:57:52.029987 7844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:57:52.030030 7844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:52.030069 7844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:52.030124 7844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:57:52.030164 7844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:57:52.035439 7844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:52.036630 7844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:52.037699 7844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:52.047021 7852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.047087 7849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.047142 7850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:52.047191 7844 server_base.cc:1034] running on GCE node
I20260806 15:57:52.047823 7844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:52.049067 7844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:52.050273 7844 hybrid_clock.cc:648] HybridClock initialized: now 1786031872050239 us; error 46 us; skew 500 ppm
Aug 06 15:57:52 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031872, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:52.059007 7844 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:57:52.060014 7844 webserver.cc:460] Webserver started at http://127.13.223.62:37483/ using document root <none> and password file <none>
I20260806 15:57:52.060746 7844 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:52.060849 7844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:52.064790 7844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:57:52.067021 7863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.068287 7844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:57:52.068429 7844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "3512c49ae4c84ea8ba597e929d1881e7"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "52536ebc6bc9f468d1685431befd870b"
server_key_iv: "dccc15f6abc55dabfd8e7ee2b39d26e3"
server_key_version: "encryptionkey@0"
I20260806 15:57:52.069073 7844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:52.092913 7844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:52.096423 7844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:52.096680 7844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:52.104384 7844 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:44879
I20260806 15:57:52.104708 7957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:44879 every 8 connection(s)
I20260806 15:57:52.105614 7844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:57:52.108946 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 7844
I20260806 15:57:52.109440 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:44383
--local_ip_for_outbound_sockets=127.13.223.1
--tserver_master_addrs=127.13.223.62:44879
--webserver_port=44333
--webserver_interface=127.13.223.1
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:57:52.110224 7959 sys_catalog.cc:263] Verifying existing consensus state
I20260806 15:57:52.111953 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap starting.
I20260806 15:57:52.130977 7959 log.cc:824] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:52.142153 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:57:52.142614 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Bootstrap complete.
I20260806 15:57:52.146874 7959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:52.147287 7959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3512c49ae4c84ea8ba597e929d1881e7, State: Initialized, Role: FOLLOWER
I20260806 15:57:52.147780 7959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:52.147926 7959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:57:52.148012 7959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:57:52.148138 7959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 15:57:52.150187 7959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:52.150521 7959 leader_election.cc:302] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3512c49ae4c84ea8ba597e929d1881e7; no voters:
I20260806 15:57:52.150844 7959 leader_election.cc:288] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 15:57:52.150986 7968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 15:57:52.151235 7968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [term 3 LEADER]: Becoming Leader. State: Replica: 3512c49ae4c84ea8ba597e929d1881e7, State: Running, Role: LEADER
I20260806 15:57:52.151598 7968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } }
I20260806 15:57:52.152103 7959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:57:52.153146 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3512c49ae4c84ea8ba597e929d1881e7. Latest consensus state: current_term: 3 leader_uuid: "3512c49ae4c84ea8ba597e929d1881e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } } }
I20260806 15:57:52.153290 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:57:52.153607 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3512c49ae4c84ea8ba597e929d1881e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3512c49ae4c84ea8ba597e929d1881e7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44879 } } }
I20260806 15:57:52.153666 7986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:57:52.153719 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:57:52.159238 7986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:57:52.159994 7986 catalog_manager.cc:1259] Loaded cluster ID: e7f3e8eeb83e4b23a4cb0343d42ad37e
I20260806 15:57:52.160075 7986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:57:52.162493 7986 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:57:52.163234 7986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Loaded TSK: 0
I20260806 15:57:52.163331 7986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3512c49ae4c84ea8ba597e929d1881e7: Loaded TSK: 1
F20260806 15:57:52.164076 7986 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f93947be9ad google::LogMessage::Fail() at ??:0
@ 0x7f93947c188a google::LogMessage::SendToLog() at ??:0
@ 0x7f93947be4de google::LogMessage::Flush() at ??:0
@ 0x7f93947bfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f939748218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f939747af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f939f1277bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f939f10199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f939f102178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f939f143df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f939f16282b std::function<>::operator()() at ??:0
@ 0x7f939f101b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f939f102f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f939f0ff579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f939f143640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9396a103d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9396a10c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9396a123f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9396a01512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f93980456db start_thread at ??:0
@ 0x7f93932bc61f clone at ??:0
W20260806 15:57:52.267993 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:52.268244 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:52.268337 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:52.272563 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:52.272648 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:52.272744 7966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:57:52.278142 7966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:44383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=44333
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:52.279228 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:52.280289 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:52.290351 8031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.290351 8030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:52.290669 7966 server_base.cc:1034] running on GCE node
W20260806 15:57:52.290367 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:52.291149 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:52.291831 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:52.293022 7966 hybrid_clock.cc:648] HybridClock initialized: now 1786031872292993 us; error 44 us; skew 500 ppm
Aug 06 15:57:52 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031872, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:52.296061 7966 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:57:52.297076 7966 webserver.cc:460] Webserver started at http://127.13.223.1:44333/ using document root <none> and password file <none>
I20260806 15:57:52.297646 7966 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:52.297744 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:52.301630 7966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:57:52.303766 8047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.305148 7966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:57:52.305243 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "0850a6fb59f54377a17cb94a872bc6b9"
format_stamp: "Formatted at 2026-08-06 15:57:09 on dist-test-slave-rwrg"
server_key: "66aaac3d2e0178aff97b23c6df014859"
server_key_iv: "649cb6e95eb9183e41b262c51ec9a547"
server_key_version: "encryptionkey@0"
I20260806 15:57:52.305794 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:52.320173 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:52.323542 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:52.323746 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:52.324361 7966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:52.325583 8055 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 15:57:52.329807 7966 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 15:57:52.329872 7966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 15:57:52.329922 7966 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 15:57:52.331625 7966 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 15:57:52.331686 7966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 15:57:52.331938 8055 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap starting.
I20260806 15:57:52.343135 7966 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:44383
I20260806 15:57:52.343416 8228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:44383 every 8 connection(s)
I20260806 15:57:52.344617 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:57:52.347749 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 7966
I20260806 15:57:52.348402 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:38851
--local_ip_for_outbound_sockets=127.13.223.2
--tserver_master_addrs=127.13.223.62:44879
--webserver_port=36785
--webserver_interface=127.13.223.2
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:57:52.355001 8055 log.cc:824] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Log is configured to *not* fsync() on all Append() calls
I20260806 15:57:52.362223 8055 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 15:57:52.362644 8055 tablet_bootstrap.cc:492] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Bootstrap complete.
I20260806 15:57:52.363878 8055 ts_tablet_manager.cc:1397] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s
I20260806 15:57:52.366463 8055 raft_consensus.cc:348] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:52.366927 8055 raft_consensus.cc:729] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0850a6fb59f54377a17cb94a872bc6b9, State: Initialized, Role: FOLLOWER
I20260806 15:57:52.367501 8055 consensus_queue.cc:260] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:52.368594 8055 ts_tablet_manager.cc:1428] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260806 15:57:52.414856 8232 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:57:52.347685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:42033)
0806 15:57:52.349355 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.349426 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:52.350538 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:52.413880 (+ 63342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1366,"thread_start_us":165,"threads_started":1}
W20260806 15:57:52.415529 8229 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:44879 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.62:44879: Client connection negotiation failed: client connection to 127.13.223.62:44879: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 15:57:52.416627 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.416209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:54665)
0806 15:57:52.416391 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.416522 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 15:57:52.417393 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.417159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:49435)
0806 15:57:52.417283 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.417335 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 15:57:52.417582 8229 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 15:57:52.493590 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:52.493853 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:52.493901 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:52.498207 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:52.498262 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:52.498324 8234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:57:52.503660 8234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:38851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=36785
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:52.504739 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:52.505836 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:52.515640 8251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.515688 8254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.515646 8248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:52.515789 8234 server_base.cc:1034] running on GCE node
I20260806 15:57:52.516335 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:52.517025 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:52.518196 8234 hybrid_clock.cc:648] HybridClock initialized: now 1786031872518164 us; error 42 us; skew 500 ppm
Aug 06 15:57:52 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031872, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:52.521106 8234 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:57:52.522037 8234 webserver.cc:460] Webserver started at http://127.13.223.2:36785/ using document root <none> and password file <none>
I20260806 15:57:52.522555 8234 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:52.522615 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:52.526499 8234 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:57:52.528760 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.530129 8234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:57:52.530352 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "fcf18a36d1f34a6eac2a4b000541ba8a"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "07f05af2ccbce7ec01f4aacd945ed4d2"
server_key_iv: "50f38795ec6661a34bf0336161db6f1f"
server_key_version: "encryptionkey@0"
I20260806 15:57:52.530902 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:52.565521 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:52.569792 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:52.570012 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:52.570828 8234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:52.572088 8234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:52.572214 8234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.572304 8234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:52.572368 8234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.590456 8234 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:38851
I20260806 15:57:52.590476 8447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:38851 every 8 connection(s)
I20260806 15:57:52.591859 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:57:52.598217 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.595613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:57835)
0806 15:57:52.596303 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.596474 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":147,"threads_started":1}
W20260806 15:57:52.598747 8452 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:44879 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.62:44879: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
I20260806 15:57:52.599661 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.599344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:55313)
0806 15:57:52.599526 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.599584 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 15:57:52.600389 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.600136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:48365)
0806 15:57:52.600258 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.600317 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 15:57:52.600610 8452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 15:57:52.601734 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 8234
I20260806 15:57:52.602322 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:37433
--local_ip_for_outbound_sockets=127.13.223.3
--tserver_master_addrs=127.13.223.62:44879
--webserver_port=36377
--webserver_interface=127.13.223.3
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:32897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:57:52.728116 8455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:57:52.728374 8455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:57:52.728420 8455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:57:52.732431 8455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:57:52.732547 8455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:57:52.732685 8455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:57:52.738107 8455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:32897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:37433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=36377
--tserver_master_addrs=127.13.223.62:44879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:57:52.739153 8455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:57:52.740159 8455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:57:52.749995 8464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.750041 8465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:57:52.750005 8468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:57:52.750247 8455 server_base.cc:1034] running on GCE node
I20260806 15:57:52.750788 8455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:57:52.751468 8455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:57:52.752609 8455 hybrid_clock.cc:648] HybridClock initialized: now 1786031872752593 us; error 27 us; skew 500 ppm
Aug 06 15:57:52 dist-test-slave-rwrg krb5kdc[6093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031872, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:57:52.755427 8455 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:57:52.756404 8455 webserver.cc:460] Webserver started at http://127.13.223.3:36377/ using document root <none> and password file <none>
I20260806 15:57:52.757004 8455 fs_manager.cc:359] Metadata directory not provided
I20260806 15:57:52.757074 8455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:57:52.760942 8455 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:57:52.763023 8477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.764248 8455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:57:52.764370 8455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "3fa0d4ea31754ef6b93440b1900d1da1"
format_stamp: "Formatted at 2026-08-06 15:57:10 on dist-test-slave-rwrg"
server_key: "a7a8051c2fcaa10184334ac849a6c544"
server_key_iv: "b50112d4f39d0046d447919bacfd12c7"
server_key_version: "encryptionkey@0"
I20260806 15:57:52.764977 8455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:57:52.795951 8455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:57:52.799352 8455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:57:52.799582 8455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:57:52.800278 8455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:57:52.801203 8455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:57:52.801254 8455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.801324 8455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:57:52.801368 8455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:57:52.811415 8455 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:37433
I20260806 15:57:52.811439 8671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:37433 every 8 connection(s)
I20260806 15:57:52.812553 8455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:57:52.816360 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.814633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:51159)
0806 15:57:52.815095 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.815215 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
W20260806 15:57:52.816949 8672 heartbeater.cc:644] Failed to heartbeat to 127.13.223.62:44879 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.62:44879: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
I20260806 15:57:52.817656 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.817411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:39351)
0806 15:57:52.817536 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.817589 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 15:57:52.818337 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:52.818124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:42891)
0806 15:57:52.818234 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:52.818279 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 15:57:52.818588 8672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 15:57:52.822275 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 8455
I20260806 15:57:53.418665 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:53.418211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:42093)
0806 15:57:53.418473 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:53.418535 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260806 15:57:53.601584 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:53.601162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:55015)
0806 15:57:53.601417 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:53.601469 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260806 15:57:53.819530 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:53.819124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:53873)
0806 15:57:53.819383 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:53.819433 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260806 15:57:54.001230 8683 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:54.001475 8683 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:54.002708 8683 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433), fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
Aug 06 15:57:54 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031872, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.3@KRBTEST.COM
Aug 06 15:57:54 dist-test-slave-rwrg krb5kdc[6093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031872, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.2@KRBTEST.COM
I20260806 15:57:54.020145 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.002898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37433 (local address 127.13.223.1:59199)
0806 15:57:54.003251 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:54.003269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:54.003381 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:54.005502 (+ 2121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:54.005509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:54.006173 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:54.007038 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:54.007061 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.009036 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.009040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:54.010764 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:54.010777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.011798 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.011802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:54.012083 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:54.012943 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:54.012970 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:54.015125 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:54.017570 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:54.017579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:54.017596 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:54.017944 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:54.018322 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:54.018325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:54.018327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:54.018482 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:54.019128 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:54.019131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:54.019392 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 15:57:54.019591 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 15:57:54.019942 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":446,"spinlock_wait_cycles":22400,"thread_start_us":81,"threads_started":1}
I20260806 15:57:54.020157 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.003121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:59199 (local address 127.13.223.3:37433)
0806 15:57:54.003480 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:54.003510 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:54.003568 (+ 58us) server_negotiation.cc:407] Connection header received
0806 15:57:54.004603 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:54.004610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:54.005034 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:54.005382 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:54.007242 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.008868 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.010903 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.011670 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.011825 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:54.015412 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:54.015418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:54.015431 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:54.015459 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:54.017426 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:54.018090 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:54.018095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:54.018099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:54.018215 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:54.018581 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:54.018587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:54.018592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:54.018999 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:54.019145 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:54.019727 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 15:57:54.019981 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260806 15:57:54.020186 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.003232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:47353 (local address 127.13.223.2:38851)
0806 15:57:54.003621 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0806 15:57:54.003643 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 15:57:54.003679 (+ 36us) server_negotiation.cc:407] Connection header received
0806 15:57:54.004559 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:57:54.004569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:57:54.005035 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:57:54.005382 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:57:54.007242 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.008868 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.010903 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.011644 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.011825 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:54.015600 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:57:54.015605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:57:54.015617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:57:54.015644 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:57:54.017518 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:54.018090 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:54.018095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:54.018099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:54.018219 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:57:54.018601 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:57:54.018606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:57:54.018608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:57:54.018996 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:57:54.019145 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 15:57:54.019703 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 15:57:54.019987 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":76,"threads_started":1}
I20260806 15:57:54.020201 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.003046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38851 (local address 127.13.223.1:47353)
0806 15:57:54.003550 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:54.003562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:57:54.003679 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:57:54.005502 (+ 1823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:57:54.005509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:57:54.006173 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:57:54.007038 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:54.007078 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.009036 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.009040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:54.010764 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:57:54.010777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:57:54.011798 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:57:54.011802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:57:54.012088 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:57:54.012943 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:57:54.012983 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:57:54.015401 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:57:54.017654 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:54.017657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:54.017659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:54.017958 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:54.018326 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:57:54.018329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:57:54.018331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:57:54.018497 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:57:54.019124 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:57:54.019130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:57:54.019392 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 15:57:54.019588 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 15:57:54.020029 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":590,"spinlock_wait_cycles":64384,"thread_start_us":86,"threads_started":1}
W20260806 15:57:54.022904 8050 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
W20260806 15:57:54.023075 8049 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:54.023192 8049 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0850a6fb59f54377a17cb94a872bc6b9; no voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a
I20260806 15:57:54.023490 8683 raft_consensus.cc:2726] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:57:54.419754 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.419463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:59669)
0806 15:57:54.419593 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:54.419663 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 15:57:54.602744 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.602369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:36165)
0806 15:57:54.602605 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:54.602660 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260806 15:57:54.820777 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:54.820253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:38265)
0806 15:57:54.820621 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:54.820685 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20260806 15:57:55.421474 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:55.420830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:53735)
0806 15:57:55.421245 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:55.421313 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":181,"threads_started":1}
I20260806 15:57:55.603855 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:55.603416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:55441)
0806 15:57:55.603700 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:55.603760 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260806 15:57:55.821879 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:55.821453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:49675)
0806 15:57:55.821740 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:55.821793 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20260806 15:57:55.828202 8702 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:55.828364 8702 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:55.828903 8702 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433), fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
W20260806 15:57:55.829743 8049 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
W20260806 15:57:55.829849 8050 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:55.830065 8050 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0850a6fb59f54377a17cb94a872bc6b9; no voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a
I20260806 15:57:55.830236 8702 raft_consensus.cc:2726] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:57:56.422834 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:56.422321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:43081)
0806 15:57:56.422622 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:56.422687 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260806 15:57:56.605293 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:56.604875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:49231)
0806 15:57:56.605147 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:56.605203 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260806 15:57:56.823045 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:56.822588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:49159)
0806 15:57:56.822892 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:56.822955 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":124,"threads_started":1}
I20260806 15:57:57.424489 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:57.423933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:34077)
0806 15:57:57.424315 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:57.424388 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":80,"threads_started":1}
I20260806 15:57:57.526099 8709 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:57.526232 8709 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:57.526738 8709 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433), fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
W20260806 15:57:57.527567 8050 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
W20260806 15:57:57.527707 8049 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:57.527784 8049 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0850a6fb59f54377a17cb94a872bc6b9; no voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a
I20260806 15:57:57.527937 8709 raft_consensus.cc:2726] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:57:57.606580 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:57.606107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:51143)
0806 15:57:57.606374 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:57.606430 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260806 15:57:57.824301 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:57.823829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:55089)
0806 15:57:57.824142 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:57.824206 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
I20260806 15:57:58.426079 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:58.425513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:58245)
0806 15:57:58.425849 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:58.425935 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":87,"threads_started":1}
I20260806 15:57:58.607792 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:58.607307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:33909)
0806 15:57:58.607624 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:58.607691 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260806 15:57:58.825485 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:58.825010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:48567)
0806 15:57:58.825296 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:58.825377 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260806 15:57:59.131765 8721 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:57:59.131904 8721 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:57:59.132392 8721 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433), fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
W20260806 15:57:59.133288 8049 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
W20260806 15:57:59.133383 8050 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
I20260806 15:57:59.133497 8050 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0850a6fb59f54377a17cb94a872bc6b9; no voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a
I20260806 15:57:59.133689 8721 raft_consensus.cc:2726] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:57:59.427491 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:59.427029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:56475)
0806 15:57:59.427326 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:59.427390 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260806 15:57:59.609495 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:59.608763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:38507)
0806 15:57:59.609216 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:59.609314 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":159,"threads_started":1}
I20260806 15:57:59.827008 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:57:59.826411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:45847)
0806 15:57:59.826829 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 15:57:59.826903 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":172,"threads_started":1}
I20260806 15:58:00.428962 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:00.428360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:39595)
0806 15:58:00.428745 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:00.428832 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":163,"threads_started":1}
I20260806 15:58:00.611426 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:00.610322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:34073)
0806 15:58:00.611273 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:00.611331 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":783,"threads_started":1}
I20260806 15:58:00.828357 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:00.827889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:33583)
0806 15:58:00.828200 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:00.828262 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
I20260806 15:58:01.430547 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:01.430028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:59625)
0806 15:58:01.430361 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:01.430432 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":118,"threads_started":1}
I20260806 15:58:01.612811 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:01.612293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:42825)
0806 15:58:01.612630 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:01.612693 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":89,"threads_started":1}
I20260806 15:58:01.829756 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:01.829317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:47299)
0806 15:58:01.829587 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:01.829660 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260806 15:58:02.431859 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:02.431413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:49859)
0806 15:58:02.431700 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:02.431762 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260806 15:58:02.615134 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:02.613717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:45899)
0806 15:58:02.614947 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:02.615001 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":1058,"threads_started":1}
I20260806 15:58:02.695876 8736 raft_consensus.cc:482] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:02.696024 8736 raft_consensus.cc:504] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa0d4ea31754ef6b93440b1900d1da1" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37433 } } peers { permanent_uuid: "fcf18a36d1f34a6eac2a4b000541ba8a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38851 } } peers { permanent_uuid: "0850a6fb59f54377a17cb94a872bc6b9" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 44383 } }
I20260806 15:58:02.696489 8736 leader_election.cc:288] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433), fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851)
W20260806 15:58:02.697347 8050 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fa0d4ea31754ef6b93440b1900d1da1 (127.13.223.3:37433): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
W20260806 15:58:02.697429 8049 leader_election.cc:341] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fcf18a36d1f34a6eac2a4b000541ba8a (127.13.223.2:38851): Not found: Tablet not found: d65f451c1907442fbcffbc74d9a01167
I20260806 15:58:02.697481 8050 leader_election.cc:302] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0850a6fb59f54377a17cb94a872bc6b9; no voters: 3fa0d4ea31754ef6b93440b1900d1da1, fcf18a36d1f34a6eac2a4b000541ba8a
I20260806 15:58:02.697605 8736 raft_consensus.cc:2726] T d65f451c1907442fbcffbc74d9a01167 P 0850a6fb59f54377a17cb94a872bc6b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 15:58:02.831010 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:02.830562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:40605)
0806 15:58:02.830855 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:02.830913 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260806 15:58:03.433207 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:03.432721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:56081)
0806 15:58:03.433022 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:03.433102 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":75,"threads_started":1}
I20260806 15:58:03.616413 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:03.615965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:51593)
0806 15:58:03.616246 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:03.616311 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260806 15:58:03.832346 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:03.831787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:39515)
0806 15:58:03.832139 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:03.832224 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":79,"threads_started":1}
I20260806 15:58:04.434546 8744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:04.434080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:36921)
0806 15:58:04.434372 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:04.434448 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260806 15:58:04.617847 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:04.617312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:53805)
0806 15:58:04.617651 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:04.617748 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
I20260806 15:58:04.833550 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:04.833113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:59357)
0806 15:58:04.833383 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:04.833447 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260806 15:58:05.435933 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:05.435436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:50975)
0806 15:58:05.435769 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:05.435834 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260806 15:58:05.619244 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:05.618749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:58709)
0806 15:58:05.619077 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:05.619148 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260806 15:58:05.835181 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:05.834596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:45217)
0806 15:58:05.834975 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:05.835071 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":91,"threads_started":1}
I20260806 15:58:06.437198 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:06.436760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:33155)
0806 15:58:06.437040 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:06.437097 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260806 15:58:06.620668 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:06.620117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:56027)
0806 15:58:06.620424 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:06.620485 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":94,"threads_started":1}
I20260806 15:58:06.836619 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:06.836107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:45645)
0806 15:58:06.836400 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:06.836460 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260806 15:58:07.438707 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:07.438225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.1:33169)
0806 15:58:07.438557 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:07.438609 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":146,"threads_started":1}
I20260806 15:58:07.622133 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:07.621662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.2:56689)
0806 15:58:07.621968 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:07.622034 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260806 15:58:07.829891 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 7966
I20260806 15:58:07.837983 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:07.837513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44879 (local address 127.13.223.3:49819)
0806 15:58:07.837789 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:07.837849 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.62:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
I20260806 15:58:07.843564 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 8234
I20260806 15:58:07.876657 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 8455
2026-08-06T15:58:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59820 ms)
[----------] 30 tests from SecurityITest (255638 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:07 dist-test-slave-rwrg krb5kdc[8779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:07 dist-test-slave-rwrg krb5kdc[8779](info): set up 2 sockets
Aug 06 15:58:07 dist-test-slave-rwrg krb5kdc[8779](info): commencing operation
krb5kdc: starting...
W20260806 15:58:09.148417 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:09 dist-test-slave-rwrg krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:09.279163 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40955
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:40955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:09.392553 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:09.392858 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:09.392942 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:09.396869 8800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:09.396950 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:09.397004 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:09.397061 8800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:09.397092 8800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:09.402347 8800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:40955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:09.403448 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:09.404577 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:09.413789 8810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:09.413846 8816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:09.413919 8800 server_base.cc:1034] running on GCE node
W20260806 15:58:09.413846 8820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:09.414561 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:09.415740 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:09.416922 8800 hybrid_clock.cc:648] HybridClock initialized: now 1786031889416892 us; error 42 us; skew 500 ppm
Aug 06 15:58:09 dist-test-slave-rwrg krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:09.419760 8800 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:09.420707 8800 webserver.cc:460] Webserver started at http://127.13.223.62:46781/ using document root <none> and password file <none>
I20260806 15:58:09.421291 8800 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:09.421388 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:09.421627 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:09.423650 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "b6b9f68277b74718aeca72a34ae2c737"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "23ea691ee668a67a54044863b863d19f"
server_key_iv: "c3c2327fab83b2ea64ac6243f9718829"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.424220 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "b6b9f68277b74718aeca72a34ae2c737"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "23ea691ee668a67a54044863b863d19f"
server_key_iv: "c3c2327fab83b2ea64ac6243f9718829"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.428011 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:58:09.430420 8828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.431574 8800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 15:58:09.431715 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "b6b9f68277b74718aeca72a34ae2c737"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "23ea691ee668a67a54044863b863d19f"
server_key_iv: "c3c2327fab83b2ea64ac6243f9718829"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.431823 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:09.442905 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:09.445847 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:09.446044 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:09.453017 8800 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:40955
I20260806 15:58:09.453049 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:40955 every 8 connection(s)
I20260806 15:58:09.453992 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:09.456004 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 8800
I20260806 15:58:09.456147 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:09.456429 14204 external_mini_cluster.cc:1428] Setting key 09c04334cc428c507e2e62499249fbb5
I20260806 15:58:09.457226 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:09.463950 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737: Bootstrap starting.
Aug 06 15:58:09 dist-test-slave-rwrg krb5kdc[8779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:09.466841 8926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:09.467800 8926 log.cc:824] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:09.470304 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737: No bootstrap required, opened a new log
I20260806 15:58:09.471411 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:09.457823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.13.223.62:40955)
0806 15:58:09.458244 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:09.458260 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:09.458295 (+ 35us) server_negotiation.cc:407] Connection header received
0806 15:58:09.458907 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:09.458923 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:09.459241 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:09.459516 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:09.460452 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:09.461513 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:09.462733 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:09.463197 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:09.463335 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:09.466127 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:09.466134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:09.466151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:09.466184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:09.468155 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:09.468808 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:09.468818 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:09.468827 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:09.468966 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:09.469233 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:09.469236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:09.469238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:09.469747 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:09.469933 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:09.470203 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 15:58:09.470495 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":155,"threads_started":1}
I20260806 15:58:09.475915 8926 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b9f68277b74718aeca72a34ae2c737" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40955 } }
I20260806 15:58:09.476234 8926 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:09.476302 8926 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b9f68277b74718aeca72a34ae2c737, State: Initialized, Role: FOLLOWER
I20260806 15:58:09.476859 8926 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b9f68277b74718aeca72a34ae2c737" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40955 } }
I20260806 15:58:09.477067 8926 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:09.477208 8926 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:09.477341 8926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:09.478353 8926 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b9f68277b74718aeca72a34ae2c737" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40955 } }
I20260806 15:58:09.478755 8926 leader_election.cc:302] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6b9f68277b74718aeca72a34ae2c737; no voters:
I20260806 15:58:09.479139 8926 leader_election.cc:288] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:09.479281 8932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:09.479543 8932 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [term 1 LEADER]: Becoming Leader. State: Replica: b6b9f68277b74718aeca72a34ae2c737, State: Running, Role: LEADER
I20260806 15:58:09.479903 8932 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b9f68277b74718aeca72a34ae2c737" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40955 } }
I20260806 15:58:09.480480 8926 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:09.482029 8933 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6b9f68277b74718aeca72a34ae2c737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b9f68277b74718aeca72a34ae2c737" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40955 } } }
I20260806 15:58:09.482138 8933 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:09.482486 8943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:09.483258 8934 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6b9f68277b74718aeca72a34ae2c737. Latest consensus state: current_term: 1 leader_uuid: "b6b9f68277b74718aeca72a34ae2c737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b9f68277b74718aeca72a34ae2c737" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40955 } } }
I20260806 15:58:09.483348 8934 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:09.485337 8943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:09.490773 8943 catalog_manager.cc:1347] Generated new cluster ID: 0e6a6d9d3db54402b9ce65f847d5fc68
I20260806 15:58:09.490839 8943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:09.503391 8943 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:09.504863 8943 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:09.522112 8943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737: Generated new TSK 0
I20260806 15:58:09.523049 8943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:58:09.564194 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40955
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:09.681891 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:09.682237 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:09.682323 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:09.686444 8961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:09.686534 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:09.686625 8961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:58:09.691988 8961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:09.693097 8961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:09.694157 8961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:09.703985 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:09.703933 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:09.704125 8961 server_base.cc:1034] running on GCE node
W20260806 15:58:09.704319 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:09.705001 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:09.705869 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:09.707034 8961 hybrid_clock.cc:648] HybridClock initialized: now 1786031889707009 us; error 43 us; skew 500 ppm
Aug 06 15:58:09 dist-test-slave-rwrg krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:09.711299 8961 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:58:09.712416 8961 webserver.cc:460] Webserver started at http://127.13.223.1:35141/ using document root <none> and password file <none>
I20260806 15:58:09.713006 8961 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:09.713076 8961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:09.713243 8961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:09.715322 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "72d282a4ada94fc8ab438cb9e622e4c2"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "8739933cd0f904501316d6c24d5d8ca4"
server_key_iv: "306ddd72a48993be5e9f85a04d3799de"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.715873 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "72d282a4ada94fc8ab438cb9e622e4c2"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "8739933cd0f904501316d6c24d5d8ca4"
server_key_iv: "306ddd72a48993be5e9f85a04d3799de"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.719911 8961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:58:09.722595 8989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.723837 8961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:09.723981 8961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "72d282a4ada94fc8ab438cb9e622e4c2"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "8739933cd0f904501316d6c24d5d8ca4"
server_key_iv: "306ddd72a48993be5e9f85a04d3799de"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.724103 8961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:09.749197 8961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:09.752701 8961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:09.752923 8961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:09.753567 8961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:09.754446 8961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:09.754518 8961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.754586 8961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:09.754642 8961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.763993 8961 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:36977
I20260806 15:58:09.764019 9166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:36977 every 8 connection(s)
I20260806 15:58:09.765056 8961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:58:09.771921 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 8961
I20260806 15:58:09.772061 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:58:09.772398 14204 external_mini_cluster.cc:1428] Setting key ad13b916fad32e7a393cfce86777a68e
Aug 06 15:58:09 dist-test-slave-rwrg krb5kdc[8779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:09.780148 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:09.767042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:58587 (local address 127.13.223.62:40955)
0806 15:58:09.767200 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:09.767205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:09.768116 (+ 911us) server_negotiation.cc:407] Connection header received
0806 15:58:09.769617 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:09.769620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:09.769668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:09.769745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:09.770982 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:09.771518 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:09.773168 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:09.773427 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:09.773514 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:09.776427 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:09.776431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:09.776433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:09.776457 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:09.777993 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:09.778563 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:09.778566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:09.778568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:09.778654 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:09.779008 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:09.779013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:09.779015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:09.779211 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:09.779323 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:09.779833 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 15:58:09.779966 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:58:09.781148 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:09.767459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40955 (local address 127.13.223.1:58587)
0806 15:58:09.767959 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:09.768003 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:09.769378 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:09.769917 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:09.769924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:09.770322 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:09.770810 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:09.770823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:09.771701 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:09.771705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:09.773049 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:09.773060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:09.773695 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:09.773699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:09.773894 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:09.774511 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:09.774534 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:09.776249 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:09.778159 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:09.778165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:09.778181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:09.778452 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:09.778753 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:09.778756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:09.778758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:09.778909 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:09.779324 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:09.779332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:09.779593 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 15:58:09.779793 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 15:58:09.780197 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":146,"threads_started":1}
I20260806 15:58:09.782747 9167 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40955
I20260806 15:58:09.783004 9167 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:58:09.783533 9167 heartbeater.cc:511] Master 127.13.223.62:40955 requested a full tablet report, sending...
I20260806 15:58:09.785187 8861 ts_manager.cc:194] Registered new tserver with Master: 72d282a4ada94fc8ab438cb9e622e4c2 (127.13.223.1:36977)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:58:09.786433 8861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:58587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:58:09.801690 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40955
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:09.917013 9175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:09.917271 9175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:09.917313 9175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:09.921924 9175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:09.922001 9175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:09.922063 9175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:58:09.927615 9175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:09.928928 9175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:09.929951 9175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:09.939975 9182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:09.939958 9181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:09.940189 9175 server_base.cc:1034] running on GCE node
W20260806 15:58:09.939958 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:09.940775 9175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:09.941522 9175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:09.942734 9175 hybrid_clock.cc:648] HybridClock initialized: now 1786031889942695 us; error 51 us; skew 500 ppm
Aug 06 15:58:09 dist-test-slave-rwrg krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:09.945818 9175 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:58:09.946779 9175 webserver.cc:460] Webserver started at http://127.13.223.2:44611/ using document root <none> and password file <none>
I20260806 15:58:09.947363 9175 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:09.947459 9175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:09.947698 9175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:09.949949 9175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "fe6deec79e13464385ff8c710740b1ae"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "8d6d9e7376d3d1dd1623b68d4b3067a0"
server_key_iv: "af1ab400be1d7488a7bdb97de2ff4d34"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.950551 9175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "fe6deec79e13464385ff8c710740b1ae"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "8d6d9e7376d3d1dd1623b68d4b3067a0"
server_key_iv: "af1ab400be1d7488a7bdb97de2ff4d34"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.954553 9175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:09.957158 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.958515 9175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:09.958652 9175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "fe6deec79e13464385ff8c710740b1ae"
format_stamp: "Formatted at 2026-08-06 15:58:09 on dist-test-slave-rwrg"
server_key: "8d6d9e7376d3d1dd1623b68d4b3067a0"
server_key_iv: "af1ab400be1d7488a7bdb97de2ff4d34"
server_key_version: "encryptionkey@0"
I20260806 15:58:09.958771 9175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:09.978118 9175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:09.981513 9175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:09.981775 9175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:09.982520 9175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:09.983515 9175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:09.983596 9175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.983704 9175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:09.983760 9175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:09.994174 9175 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:44679
I20260806 15:58:09.994195 9356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:44679 every 8 connection(s)
I20260806 15:58:09.995149 9175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:58:09.999269 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 9175
I20260806 15:58:09.999430 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:58:09.999765 14204 external_mini_cluster.cc:1428] Setting key a747b4595cf9fbf73c099ca7611a4d8a
Aug 06 15:58:10 dist-test-slave-rwrg krb5kdc[8779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031889, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:10.009907 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:09.997254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:59307 (local address 127.13.223.62:40955)
0806 15:58:09.997434 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:09.997439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:09.998198 (+ 759us) server_negotiation.cc:407] Connection header received
0806 15:58:09.999551 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:09.999555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:09.999613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:09.999688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:10.001113 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.001753 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.003023 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.003255 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.003362 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.006047 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:10.006050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:10.006052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:10.006077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:10.007652 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:10.008195 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:10.008203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:10.008204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:10.008283 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:10.008707 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:10.008711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:10.008713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:10.008902 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:10.009039 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.009607 (+ 568us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.009738 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:58:10.010887 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:09.997558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40955 (local address 127.13.223.2:59307)
0806 15:58:09.998032 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:09.998077 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:09.999306 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:09.999881 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:09.999892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.000376 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:10.000960 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.000974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.001953 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.001956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.002880 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.002891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.003420 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.003424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.003605 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.004240 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:10.004257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:10.005866 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:10.007775 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:10.007782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:10.007797 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:10.008066 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:10.008389 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:10.008392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:10.008394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:10.008610 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:10.009036 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:10.009044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:10.009351 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.009545 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.009890 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260806 15:58:10.012300 9357 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40955
I20260806 15:58:10.012624 9357 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:10.013154 9357 heartbeater.cc:511] Master 127.13.223.62:40955 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:58:10.014418 8861 ts_manager.cc:194] Registered new tserver with Master: fe6deec79e13464385ff8c710740b1ae (127.13.223.2:44679)
I20260806 15:58:10.015332 8861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:59307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:58:10.031266 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40955
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:10.147275 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:10.147535 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:10.147579 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:10.151620 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:10.151687 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:10.151784 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:58:10.157125 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:10.158216 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:10.159241 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:10.169631 9372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:10.169593 9371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:10.169572 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:10.169881 9365 server_base.cc:1034] running on GCE node
I20260806 15:58:10.170508 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:10.171180 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:10.172356 9365 hybrid_clock.cc:648] HybridClock initialized: now 1786031890172339 us; error 32 us; skew 500 ppm
Aug 06 15:58:10 dist-test-slave-rwrg krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:10.175446 9365 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:58:10.176559 9365 webserver.cc:460] Webserver started at http://127.13.223.3:34433/ using document root <none> and password file <none>
I20260806 15:58:10.177191 9365 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:10.177292 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:10.177556 9365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:10.179903 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "67bdca142e77407ab374d6844e27be37"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "75e26f4ea97480a6418de91b4a61c5f2"
server_key_iv: "85a11d66df607a83ee3d0bd90d7c8e56"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.180657 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "67bdca142e77407ab374d6844e27be37"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "75e26f4ea97480a6418de91b4a61c5f2"
server_key_iv: "85a11d66df607a83ee3d0bd90d7c8e56"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.185032 9365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:58:10.187783 9389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.189164 9365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:10.189345 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "67bdca142e77407ab374d6844e27be37"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "75e26f4ea97480a6418de91b4a61c5f2"
server_key_iv: "85a11d66df607a83ee3d0bd90d7c8e56"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.189471 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:10.224592 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:10.228163 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:10.228366 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:10.229094 9365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:10.229969 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:10.230046 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.230119 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:10.230166 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.240115 9365 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:37073
I20260806 15:58:10.240206 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:37073 every 8 connection(s)
I20260806 15:58:10.241117 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:58:10.249193 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 9365
I20260806 15:58:10.249377 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:58:10.249818 14204 external_mini_cluster.cc:1428] Setting key 5fc84564835eaa8c6ba7c331604befd8
Aug 06 15:58:10 dist-test-slave-rwrg krb5kdc[8779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:10.255446 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.243019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:45705 (local address 127.13.223.62:40955)
0806 15:58:10.243170 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.243174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.244535 (+ 1361us) server_negotiation.cc:407] Connection header received
0806 15:58:10.245465 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.245468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.245524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.245669 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:10.246838 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.247372 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.248625 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.248844 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.248965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.251814 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:10.251818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:10.251820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:10.251855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:10.253311 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:10.253850 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:10.253854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:10.253856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:10.253952 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:10.254308 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:10.254313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:10.254315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:10.254506 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:10.254627 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.255164 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.255290 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:58:10.256297 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.243453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:40955 (local address 127.13.223.3:45705)
0806 15:58:10.244357 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.244402 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.245257 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.245847 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.245854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.246214 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:10.246692 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.246706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.247518 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.247521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.248400 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.248410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.249022 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.249026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.249201 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.249888 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:10.249905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:10.251645 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:10.253438 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:10.253444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:10.253459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:10.253739 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:10.254049 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:10.254053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:10.254055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:10.254206 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:10.254616 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:10.254623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:10.254897 (+ 274us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.255117 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.255451 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":556,"threads_started":1}
I20260806 15:58:10.257490 9558 heartbeater.cc:347] Connected to a master server at 127.13.223.62:40955
I20260806 15:58:10.257774 9558 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:10.258317 9558 heartbeater.cc:511] Master 127.13.223.62:40955 requested a full tablet report, sending...
I20260806 15:58:10.259408 8861 ts_manager.cc:194] Registered new tserver with Master: 67bdca142e77407ab374d6844e27be37 (127.13.223.3:37073)
I20260806 15:58:10.259989 8861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:45705
I20260806 15:58:10.264063 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:58:10.274147 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.266255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.13.223.62:40955)
0806 15:58:10.266419 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.266424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.266639 (+ 215us) server_negotiation.cc:407] Connection header received
0806 15:58:10.266828 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.266832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.266883 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.266997 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:10.267815 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.268352 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.269286 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.269529 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.269650 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.271035 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:10.271039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:10.271041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:10.271072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:10.272377 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:10.272846 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:10.272851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:10.272853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:10.272933 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:10.273221 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:10.273224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:10.273226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:10.273457 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:10.273610 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.273843 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.273957 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 15:58:10.277249 8861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:58:10.280350 8861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:58:10.296175 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.290286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44679 (local address 127.0.0.1:49788)
0806 15:58:10.290539 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.290551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.290629 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.291451 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.291457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.291480 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:10.291659 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.291667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.293323 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.293326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.294770 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.294777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.295607 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.295611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.295712 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.295748 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.295865 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.295989 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":209,"thread_start_us":99,"threads_started":1}
I20260806 15:58:10.296195 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.289666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36977 (local address 127.0.0.1:33450)
0806 15:58:10.290244 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.290257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.290319 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.291565 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.291568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.291584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:10.291705 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.291712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.293167 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.293174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.294753 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.294765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.295779 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.295783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.295901 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.295926 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.295976 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.296040 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":125,"thread_start_us":102,"threads_started":1}
I20260806 15:58:10.296713 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.289628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.13.223.3:37073)
0806 15:58:10.291088 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.291100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.291125 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:58:10.291205 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.291211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.291399 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.291556 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:10.291964 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.293154 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.294673 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.295510 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.295622 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.295779 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.296409 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.296534 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":181,"threads_started":1}
I20260806 15:58:10.296205 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.290583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.13.223.2:44679)
0806 15:58:10.291023 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.291035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.291056 (+ 21us) server_negotiation.cc:407] Connection header received
0806 15:58:10.291127 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.291131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.291282 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.291463 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:10.291778 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.293153 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.294926 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.295470 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.295599 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.295756 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.295916 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.296026 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":195,"threads_started":1}
I20260806 15:58:10.296988 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.290724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.13.223.1:36977)
0806 15:58:10.291121 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.291129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.291149 (+ 20us) server_negotiation.cc:407] Connection header received
0806 15:58:10.291214 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.291218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.291399 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.291852 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:10.292020 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.292850 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.294886 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.295644 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.296271 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.296667 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.296718 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.296816 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260806 15:58:10.296227 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.289413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37073 (local address 127.0.0.1:54942)
0806 15:58:10.290118 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.290172 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.290319 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.291716 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.291719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.291735 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:10.291842 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.291848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.293323 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.293326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.294542 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.294553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.295838 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.295843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.295939 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.295959 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.296038 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.296112 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":88,"thread_start_us":223,"threads_started":1}
I20260806 15:58:10.299748 9263 tablet_service.cc:1467] Processing CreateTablet for tablet 403029de743c4c56b2c03eb4aa0eb032 (DEFAULT_TABLE table=test-table [id=f167dee958d8454daf7cb1656f9b1e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:10.299882 9463 tablet_service.cc:1467] Processing CreateTablet for tablet 403029de743c4c56b2c03eb4aa0eb032 (DEFAULT_TABLE table=test-table [id=f167dee958d8454daf7cb1656f9b1e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:10.299878 9060 tablet_service.cc:1467] Processing CreateTablet for tablet 403029de743c4c56b2c03eb4aa0eb032 (DEFAULT_TABLE table=test-table [id=f167dee958d8454daf7cb1656f9b1e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:10.300889 9263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403029de743c4c56b2c03eb4aa0eb032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:10.300951 9463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403029de743c4c56b2c03eb4aa0eb032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:10.301151 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403029de743c4c56b2c03eb4aa0eb032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:10.307464 9580 tablet_bootstrap.cc:492] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Bootstrap starting.
I20260806 15:58:10.308969 9584 tablet_bootstrap.cc:492] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Bootstrap starting.
I20260806 15:58:10.309826 9580 tablet_bootstrap.cc:654] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:10.310783 9584 tablet_bootstrap.cc:654] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:10.311189 9580 log.cc:824] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:10.311807 9583 tablet_bootstrap.cc:492] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2: Bootstrap starting.
I20260806 15:58:10.312119 9584 log.cc:824] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:10.313508 9580 tablet_bootstrap.cc:492] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: No bootstrap required, opened a new log
I20260806 15:58:10.313719 9580 ts_tablet_manager.cc:1397] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 15:58:10.313969 9583 tablet_bootstrap.cc:654] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:10.314213 9584 tablet_bootstrap.cc:492] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: No bootstrap required, opened a new log
I20260806 15:58:10.314388 9584 ts_tablet_manager.cc:1397] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:58:10.314779 9583 log.cc:824] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:10.316358 9583 tablet_bootstrap.cc:492] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2: No bootstrap required, opened a new log
I20260806 15:58:10.316584 9583 ts_tablet_manager.cc:1397] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260806 15:58:10.317772 9580 raft_consensus.cc:348] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.318084 9580 raft_consensus.cc:374] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:10.318145 9580 raft_consensus.cc:729] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6deec79e13464385ff8c710740b1ae, State: Initialized, Role: FOLLOWER
I20260806 15:58:10.318077 9584 raft_consensus.cc:348] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.318321 9584 raft_consensus.cc:374] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:10.318377 9584 raft_consensus.cc:729] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bdca142e77407ab374d6844e27be37, State: Initialized, Role: FOLLOWER
I20260806 15:58:10.318610 9580 consensus_queue.cc:260] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.318791 9584 consensus_queue.cc:260] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.319643 9584 ts_tablet_manager.cc:1428] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 15:58:10.319639 9580 ts_tablet_manager.cc:1428] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:58:10.319751 9558 heartbeater.cc:503] Master 127.13.223.62:40955 was elected leader, sending a full tablet report...
I20260806 15:58:10.319957 9357 heartbeater.cc:503] Master 127.13.223.62:40955 was elected leader, sending a full tablet report...
I20260806 15:58:10.320737 9583 raft_consensus.cc:348] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.320988 9583 raft_consensus.cc:374] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:10.321055 9583 raft_consensus.cc:729] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d282a4ada94fc8ab438cb9e622e4c2, State: Initialized, Role: FOLLOWER
I20260806 15:58:10.321494 9583 consensus_queue.cc:260] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.322176 9167 heartbeater.cc:503] Master 127.13.223.62:40955 was elected leader, sending a full tablet report...
I20260806 15:58:10.322402 9583 ts_tablet_manager.cc:1428] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:58:10.364121 9594 raft_consensus.cc:482] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:10.364392 9594 raft_consensus.cc:504] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.365908 9594 leader_election.cc:288] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d282a4ada94fc8ab438cb9e622e4c2 (127.13.223.1:36977), fe6deec79e13464385ff8c710740b1ae (127.13.223.2:44679)
I20260806 15:58:10.370604 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.366180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36977 (local address 127.13.223.3:45021)
0806 15:58:10.366333 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.366350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.366452 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.366960 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.366986 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.367013 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:10.367159 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.367168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.368400 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.368402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.369261 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.369271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.370169 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.370173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.370281 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.370309 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.370388 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.370461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 15:58:10.370656 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.366297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:45021 (local address 127.13.223.1:36977)
0806 15:58:10.366452 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.366457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.366474 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:58:10.366552 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.366556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.366616 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.366713 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:10.367318 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.368286 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.369405 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.370043 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.370151 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.370195 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.370428 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.370509 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 15:58:10.370913 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.366465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:44679 (local address 127.13.223.3:47501)
0806 15:58:10.366892 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.366907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.367125 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.367437 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.367439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.367454 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:10.367554 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.367557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.368607 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.368612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.369672 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.369683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.370441 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.370446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.370562 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.370587 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.370674 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.370753 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"spinlock_wait_cycles":1280,"thread_start_us":139,"threads_started":1}
I20260806 15:58:10.370926 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.366732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:47501 (local address 127.13.223.2:44679)
0806 15:58:10.366901 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.366905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.366993 (+ 88us) server_negotiation.cc:407] Connection header received
0806 15:58:10.367241 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.367248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.367310 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.367419 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:10.367679 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.368465 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.369800 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.370294 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.370419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.370464 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.370710 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.370801 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 15:58:10.371670 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403029de743c4c56b2c03eb4aa0eb032" candidate_uuid: "67bdca142e77407ab374d6844e27be37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6deec79e13464385ff8c710740b1ae" is_pre_election: true
I20260806 15:58:10.371668 9089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403029de743c4c56b2c03eb4aa0eb032" candidate_uuid: "67bdca142e77407ab374d6844e27be37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" is_pre_election: true
I20260806 15:58:10.371984 9089 raft_consensus.cc:2454] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bdca142e77407ab374d6844e27be37 in term 0.
I20260806 15:58:10.371994 9286 raft_consensus.cc:2454] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bdca142e77407ab374d6844e27be37 in term 0.
I20260806 15:58:10.372540 9392 leader_election.cc:302] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bdca142e77407ab374d6844e27be37, fe6deec79e13464385ff8c710740b1ae; no voters:
I20260806 15:58:10.372907 9594 raft_consensus.cc:2781] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:58:10.373020 9594 raft_consensus.cc:482] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:58:10.373078 9594 raft_consensus.cc:3037] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:10.374233 9594 raft_consensus.cc:504] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.374560 9594 leader_election.cc:288] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [CANDIDATE]: Term 1 election: Requested vote from peers 72d282a4ada94fc8ab438cb9e622e4c2 (127.13.223.1:36977), fe6deec79e13464385ff8c710740b1ae (127.13.223.2:44679)
I20260806 15:58:10.374998 9089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403029de743c4c56b2c03eb4aa0eb032" candidate_uuid: "67bdca142e77407ab374d6844e27be37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d282a4ada94fc8ab438cb9e622e4c2"
I20260806 15:58:10.375133 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403029de743c4c56b2c03eb4aa0eb032" candidate_uuid: "67bdca142e77407ab374d6844e27be37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6deec79e13464385ff8c710740b1ae"
I20260806 15:58:10.375144 9089 raft_consensus.cc:3037] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:10.375290 9286 raft_consensus.cc:3037] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:10.376327 9089 raft_consensus.cc:2454] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bdca142e77407ab374d6844e27be37 in term 1.
I20260806 15:58:10.376466 9286 raft_consensus.cc:2454] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bdca142e77407ab374d6844e27be37 in term 1.
I20260806 15:58:10.376729 9392 leader_election.cc:302] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bdca142e77407ab374d6844e27be37, 72d282a4ada94fc8ab438cb9e622e4c2; no voters:
I20260806 15:58:10.376945 9594 raft_consensus.cc:2781] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:10.377220 9594 raft_consensus.cc:686] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 1 LEADER]: Becoming Leader. State: Replica: 67bdca142e77407ab374d6844e27be37, State: Running, Role: LEADER
I20260806 15:58:10.377595 9594 consensus_queue.cc:237] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } }
I20260806 15:58:10.381186 8860 catalog_manager.cc:5495] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67bdca142e77407ab374d6844e27be37 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "67bdca142e77407ab374d6844e27be37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bdca142e77407ab374d6844e27be37" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 } health_report { overall_health: UNKNOWN } } }
I20260806 15:58:10.394778 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.390948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.13.223.3:37073)
0806 15:58:10.391089 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.391093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.391107 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:10.391268 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.391270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.391327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.391419 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:10.391630 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.392165 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.393125 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.393389 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.393475 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.393814 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:10.394380 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:10.394491 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.394559 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.394621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:58:10.401026 9089 raft_consensus.cc:1261] T 403029de743c4c56b2c03eb4aa0eb032 P 72d282a4ada94fc8ab438cb9e622e4c2 [term 1 FOLLOWER]: Refusing update from remote peer 67bdca142e77407ab374d6844e27be37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:10.401041 9286 raft_consensus.cc:1261] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 1 FOLLOWER]: Refusing update from remote peer 67bdca142e77407ab374d6844e27be37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:10.401787 9594 consensus_queue.cc:1035] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d282a4ada94fc8ab438cb9e622e4c2" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:10.402022 9599 consensus_queue.cc:1035] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6deec79e13464385ff8c710740b1ae" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:10.411772 9604 mvcc.cc:205] Tried to move back new op lower bound from 7315586623074828288 to 7315586622989488128. Current Snapshot: MvccSnapshot[applied={T|T < 7315586623074828288}]
I20260806 15:58:10.421721 8860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:10.421943 8860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:10.424441 8860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6b9f68277b74718aeca72a34ae2c737: Sending DeleteTablet for 3 replicas of tablet 403029de743c4c56b2c03eb4aa0eb032
I20260806 15:58:10.425297 9463 tablet_service.cc:1514] Processing DeleteTablet for tablet 403029de743c4c56b2c03eb4aa0eb032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:54942
I20260806 15:58:10.425383 9263 tablet_service.cc:1514] Processing DeleteTablet for tablet 403029de743c4c56b2c03eb4aa0eb032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:49788
I20260806 15:58:10.425422 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet 403029de743c4c56b2c03eb4aa0eb032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:10 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:33450
I20260806 15:58:10.425897 9616 tablet_replica.cc:330] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: stopping tablet replica
I20260806 15:58:10.426075 9619 tablet_replica.cc:330] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: stopping tablet replica
I20260806 15:58:10.426339 9616 raft_consensus.cc:2229] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:58:10.426397 9619 raft_consensus.cc:2229] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:10.426709 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 8961
I20260806 15:58:10.426734 9619 raft_consensus.cc:2258] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:10.439872 9619 ts_tablet_manager.cc:1905] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:10.440696 9616 raft_consensus.cc:2258] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:10.444402 9619 ts_tablet_manager.cc:1918] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:10.444831 9619 log.cc:1196] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/403029de743c4c56b2c03eb4aa0eb032
I20260806 15:58:10.445252 9619 ts_tablet_manager.cc:1939] T 403029de743c4c56b2c03eb4aa0eb032 P fe6deec79e13464385ff8c710740b1ae: Deleting consensus metadata
I20260806 15:58:10.447319 8833 catalog_manager.cc:4841] TS fe6deec79e13464385ff8c710740b1ae (127.13.223.2:44679): tablet 403029de743c4c56b2c03eb4aa0eb032 (table test-table [id=f167dee958d8454daf7cb1656f9b1e10]) successfully deleted
I20260806 15:58:10.456902 9616 ts_tablet_manager.cc:1905] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:10.459892 9616 ts_tablet_manager.cc:1918] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:10.460011 9616 log.cc:1196] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/403029de743c4c56b2c03eb4aa0eb032
I20260806 15:58:10.460384 9616 ts_tablet_manager.cc:1939] T 403029de743c4c56b2c03eb4aa0eb032 P 67bdca142e77407ab374d6844e27be37: Deleting consensus metadata
I20260806 15:58:10.461815 8829 catalog_manager.cc:4841] TS 67bdca142e77407ab374d6844e27be37 (127.13.223.3:37073): tablet 403029de743c4c56b2c03eb4aa0eb032 (table test-table [id=f167dee958d8454daf7cb1656f9b1e10]) successfully deleted
W20260806 15:58:10.463075 8833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:36977 (error 108)
I20260806 15:58:10.463959 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 9175
I20260806 15:58:10.464859 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.464245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36977 (local address 127.0.0.1:33460)
0806 15:58:10.464672 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.464758 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312}
W20260806 15:58:10.465058 8833 catalog_manager.cc:4584] TS 72d282a4ada94fc8ab438cb9e622e4c2 (127.13.223.1:36977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 403029de743c4c56b2c03eb4aa0eb032: Network error: Client connection negotiation failed: client connection to 127.13.223.1:36977: connect: Connection refused (error 111)
I20260806 15:58:10.477346 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 9365
I20260806 15:58:10.491734 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 8800
2026-08-06T15:58:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2618 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T15:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:10Z Disabled control of system clock
I20260806 15:58:10.532254 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43537
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 15:58:10.645589 9624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:10.645848 9624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:10.645884 9624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:10.649997 9624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:10.650053 9624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:10.650070 9624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:10.650116 9624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:10.650148 9624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:10.654910 9624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:43537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:10.655979 9624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:10.657469 9624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:10.665613 9643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:10.665649 9641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:10.665766 9624 server_base.cc:1034] running on GCE node
W20260806 15:58:10.665971 9637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:10.666594 9624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:10.667716 9624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:10.668900 9624 hybrid_clock.cc:648] HybridClock initialized: now 1786031890668888 us; error 26 us; skew 500 ppm
I20260806 15:58:10.670830 9624 webserver.cc:460] Webserver started at http://127.13.223.62:40325/ using document root <none> and password file <none>
I20260806 15:58:10.671416 9624 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:10.671509 9624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:10.671734 9624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:10.673785 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "bd17b6ad39104fae934922247c31f097"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "0c4d4246d0fd9c0e5b2d4a7002111f78"
server_key_iv: "de1550faa8572641522882b3ff93370c"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.674347 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "bd17b6ad39104fae934922247c31f097"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "0c4d4246d0fd9c0e5b2d4a7002111f78"
server_key_iv: "de1550faa8572641522882b3ff93370c"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.677888 9624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:58:10.680265 9654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.681553 9624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:10.681696 9624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "bd17b6ad39104fae934922247c31f097"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "0c4d4246d0fd9c0e5b2d4a7002111f78"
server_key_iv: "de1550faa8572641522882b3ff93370c"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.681810 9624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:10.699716 9624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:10.700479 9624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:10.700738 9624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:10.707921 9624 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:43537
I20260806 15:58:10.707929 9736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:43537 every 8 connection(s)
I20260806 15:58:10.708913 9624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:10.711987 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:10.717475 9737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097: Bootstrap starting.
I20260806 15:58:10.718488 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 9624
I20260806 15:58:10.718650 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:10.718935 14204 external_mini_cluster.cc:1428] Setting key 2667686cfad7b6247107605a283b3552
I20260806 15:58:10.720396 9737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:10.721201 9737 log.cc:824] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:10.723362 9737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097: No bootstrap required, opened a new log
I20260806 15:58:10.727718 9737 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd17b6ad39104fae934922247c31f097" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43537 } }
I20260806 15:58:10.727964 9737 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:10.728046 9737 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd17b6ad39104fae934922247c31f097, State: Initialized, Role: FOLLOWER
I20260806 15:58:10.728168 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.720113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.13.223.62:43537)
0806 15:58:10.720857 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.720871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.720906 (+ 35us) server_negotiation.cc:407] Connection header received
0806 15:58:10.721601 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.721620 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.721941 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.722220 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:10.722543 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.723514 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.724727 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.725205 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.725327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.726252 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:10.726256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:10.726269 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:10.726682 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:10.726738 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:10.726757 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:10.726832 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.727024 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.727342 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":202,"threads_started":1}
I20260806 15:58:10.728581 9737 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd17b6ad39104fae934922247c31f097" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43537 } }
I20260806 15:58:10.728747 9737 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:10.728816 9737 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:10.728912 9737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:10.729846 9737 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd17b6ad39104fae934922247c31f097" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43537 } }
I20260806 15:58:10.730216 9737 leader_election.cc:302] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd17b6ad39104fae934922247c31f097; no voters:
I20260806 15:58:10.730572 9737 leader_election.cc:288] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:10.730652 9744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:10.730859 9744 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [term 1 LEADER]: Becoming Leader. State: Replica: bd17b6ad39104fae934922247c31f097, State: Running, Role: LEADER
I20260806 15:58:10.731160 9744 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd17b6ad39104fae934922247c31f097" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43537 } }
I20260806 15:58:10.731878 9737 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:10.732908 9746 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd17b6ad39104fae934922247c31f097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd17b6ad39104fae934922247c31f097" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43537 } } }
I20260806 15:58:10.733040 9746 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:10.733559 9750 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd17b6ad39104fae934922247c31f097. Latest consensus state: current_term: 1 leader_uuid: "bd17b6ad39104fae934922247c31f097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd17b6ad39104fae934922247c31f097" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 43537 } } }
I20260806 15:58:10.734154 9750 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:10.734153 9758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:10.736912 9758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:10.741876 9758 catalog_manager.cc:1347] Generated new cluster ID: 4f015ad3669c4b3f825f9b5c68c7d7c5
I20260806 15:58:10.741937 9758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:10.754004 9758 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:10.754935 9758 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:10.760082 9758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097: Generated new TSK 0
I20260806 15:58:10.760794 9758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:10.770654 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43537
--builtin_ntp_servers=127.13.223.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 15:58:10.886054 9773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:10.886328 9773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:10.886413 9773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:10.890254 9773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:10.890303 9773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:10.890394 9773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:58:10.894999 9773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:10.896021 9773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:10.897403 9773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:10.907321 9784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:10.907426 9773 server_base.cc:1034] running on GCE node
W20260806 15:58:10.907435 9783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:10.907577 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:10.908070 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:10.908877 9773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:10.910066 9773 hybrid_clock.cc:648] HybridClock initialized: now 1786031890910046 us; error 47 us; skew 500 ppm
I20260806 15:58:10.911980 9773 webserver.cc:460] Webserver started at http://127.13.223.1:36031/ using document root <none> and password file <none>
I20260806 15:58:10.912626 9773 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:10.912693 9773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:10.912918 9773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:10.914994 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "e7fb335b8f6a9fc8b19dadf25fcb674b"
server_key_iv: "f4cd4298ddaa218dc0f9fbf9fe85a520"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.915561 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "e7fb335b8f6a9fc8b19dadf25fcb674b"
server_key_iv: "f4cd4298ddaa218dc0f9fbf9fe85a520"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.919270 9773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 15:58:10.921768 9794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.922969 9773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:10.923105 9773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d"
format_stamp: "Formatted at 2026-08-06 15:58:10 on dist-test-slave-rwrg"
server_key: "e7fb335b8f6a9fc8b19dadf25fcb674b"
server_key_iv: "f4cd4298ddaa218dc0f9fbf9fe85a520"
server_key_version: "encryptionkey@0"
I20260806 15:58:10.923216 9773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:10.940953 9773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:10.941642 9773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:10.941850 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:10.942409 9773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:10.943349 9773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:10.943419 9773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.943490 9773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:10.943540 9773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:10.952405 9773 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:36997
I20260806 15:58:10.952457 9979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:36997 every 8 connection(s)
I20260806 15:58:10.953321 9773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:58:10.958254 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 9773
I20260806 15:58:10.958392 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:58:10.958696 14204 external_mini_cluster.cc:1428] Setting key cdd11971a540b5e29bb787d875e14d61
I20260806 15:58:10.960767 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43537
--builtin_ntp_servers=127.13.223.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 15:58:10.963213 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.955187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:38893 (local address 127.13.223.62:43537)
0806 15:58:10.955336 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:10.955339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:10.956357 (+ 1018us) server_negotiation.cc:407] Connection header received
0806 15:58:10.957856 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:10.957859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:10.957916 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:10.958064 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:10.959115 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.959803 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.961136 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.961539 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.961669 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.962723 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:10.962727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:10.962729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:10.962740 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:10.962759 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:10.962766 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:10.962846 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:10.963007 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 15:58:10.963073 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 15:58:10.964046 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:10.955526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43537 (local address 127.13.223.1:38893)
0806 15:58:10.956209 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:10.956250 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:10.957631 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:10.958237 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:10.958244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:10.958350 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:10.958940 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.958955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.959965 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.959972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.961014 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:10.961025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:10.961699 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:10.961702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:10.961831 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:10.962456 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:10.962457 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:10.962495 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:10.962563 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:10.962848 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:10.962861 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:10.962869 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 15:58:10.963004 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 15:58:10.963267 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":115,"threads_started":1}
I20260806 15:58:10.965341 9980 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43537
I20260806 15:58:10.965606 9980 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:10.966302 9980 heartbeater.cc:511] Master 127.13.223.62:43537 requested a full tablet report, sending...
I20260806 15:58:10.968020 9679 ts_manager.cc:194] Registered new tserver with Master: a4bcbc6ba5ee4d0d819e5e468dc00c1d (127.13.223.1:36997)
I20260806 15:58:10.969228 9679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.1:38893
W20260806 15:58:11.074080 9983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:11.074326 9983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:11.074364 9983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:11.078421 9983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:11.078473 9983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:11.078526 9983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:58:11.083222 9983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:11.084287 9983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:11.085788 9983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:11.095305 9989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:11.095295 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:11.095445 9995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:11.095546 9983 server_base.cc:1034] running on GCE node
I20260806 15:58:11.096083 9983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:11.096778 9983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:11.097934 9983 hybrid_clock.cc:648] HybridClock initialized: now 1786031891097893 us; error 55 us; skew 500 ppm
I20260806 15:58:11.099870 9983 webserver.cc:460] Webserver started at http://127.13.223.2:46223/ using document root <none> and password file <none>
I20260806 15:58:11.100446 9983 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:11.100566 9983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:11.100790 9983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:11.102818 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "eeb02370972d4adfa4f375d7be29902f"
format_stamp: "Formatted at 2026-08-06 15:58:11 on dist-test-slave-rwrg"
server_key: "4b5d844d1207c37a214f31d4f786ad91"
server_key_iv: "cee980e4059c0ed98870b8c672e7aaf5"
server_key_version: "encryptionkey@0"
I20260806 15:58:11.103367 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "eeb02370972d4adfa4f375d7be29902f"
format_stamp: "Formatted at 2026-08-06 15:58:11 on dist-test-slave-rwrg"
server_key: "4b5d844d1207c37a214f31d4f786ad91"
server_key_iv: "cee980e4059c0ed98870b8c672e7aaf5"
server_key_version: "encryptionkey@0"
I20260806 15:58:11.107012 9983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260806 15:58:11.109582 10005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:11.110831 9983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:11.110965 9983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "eeb02370972d4adfa4f375d7be29902f"
format_stamp: "Formatted at 2026-08-06 15:58:11 on dist-test-slave-rwrg"
server_key: "4b5d844d1207c37a214f31d4f786ad91"
server_key_iv: "cee980e4059c0ed98870b8c672e7aaf5"
server_key_version: "encryptionkey@0"
I20260806 15:58:11.111074 9983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:11.141229 9983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:11.141983 9983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:11.142194 9983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:11.142840 9983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:11.143813 9983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:11.143885 9983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:11.143960 9983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:11.144006 9983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:11.153311 9983 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:40779
I20260806 15:58:11.153337 10172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:40779 every 8 connection(s)
I20260806 15:58:11.154219 9983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:58:11.158816 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 9983
I20260806 15:58:11.158951 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:58:11.159255 14204 external_mini_cluster.cc:1428] Setting key 6177ae67382de9500b651bfeddac87bb
I20260806 15:58:11.161206 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43537
--builtin_ntp_servers=127.13.223.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 15:58:11.163964 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.156120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:32817 (local address 127.13.223.62:43537)
0806 15:58:11.156273 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.156277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.157353 (+ 1076us) server_negotiation.cc:407] Connection header received
0806 15:58:11.158861 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.158864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.158947 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.159065 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:11.160065 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.160665 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.161935 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.162159 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.162271 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.163511 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:11.163514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:11.163516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:11.163528 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:11.163550 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:11.163558 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:11.163617 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.163807 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.163855 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 15:58:11.165023 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.156488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43537 (local address 127.13.223.2:32817)
0806 15:58:11.157200 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.157240 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.158614 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.159273 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.159280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.159389 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:11.159904 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.159918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.160796 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.160799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.161797 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.161807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.162314 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.162318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.162482 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.163195 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:11.163199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:11.163256 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:11.163343 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:11.163647 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:11.163658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:11.163665 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.163800 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.164128 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":123,"threads_started":1}
I20260806 15:58:11.166275 10173 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43537
I20260806 15:58:11.166512 10173 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:11.167045 10173 heartbeater.cc:511] Master 127.13.223.62:43537 requested a full tablet report, sending...
I20260806 15:58:11.168131 9679 ts_manager.cc:194] Registered new tserver with Master: eeb02370972d4adfa4f375d7be29902f (127.13.223.2:40779)
I20260806 15:58:11.168752 9679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.2:32817
W20260806 15:58:11.274499 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:11.274789 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:11.274856 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:11.278760 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:11.278846 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:11.278935 10177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:58:11.283586 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:11.284758 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:11.286242 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:11.295818 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:11.295938 10177 server_base.cc:1034] running on GCE node
W20260806 15:58:11.295758 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:11.295797 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:11.296387 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:11.297053 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:11.298250 10177 hybrid_clock.cc:648] HybridClock initialized: now 1786031891298237 us; error 23 us; skew 500 ppm
I20260806 15:58:11.300190 10177 webserver.cc:460] Webserver started at http://127.13.223.3:35125/ using document root <none> and password file <none>
I20260806 15:58:11.300841 10177 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:11.300936 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:11.301173 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:11.303156 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "91b129c472ab476993e1f82c29278007"
format_stamp: "Formatted at 2026-08-06 15:58:11 on dist-test-slave-rwrg"
server_key: "74c566f9afd78ddc67c16fa7d9d7401b"
server_key_iv: "9f086c83f6d3d57dad912a35e81ee6c7"
server_key_version: "encryptionkey@0"
I20260806 15:58:11.303725 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "91b129c472ab476993e1f82c29278007"
format_stamp: "Formatted at 2026-08-06 15:58:11 on dist-test-slave-rwrg"
server_key: "74c566f9afd78ddc67c16fa7d9d7401b"
server_key_iv: "9f086c83f6d3d57dad912a35e81ee6c7"
server_key_version: "encryptionkey@0"
I20260806 15:58:11.307297 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260806 15:58:11.309774 10200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:11.310938 10177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:11.311076 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "91b129c472ab476993e1f82c29278007"
format_stamp: "Formatted at 2026-08-06 15:58:11 on dist-test-slave-rwrg"
server_key: "74c566f9afd78ddc67c16fa7d9d7401b"
server_key_iv: "9f086c83f6d3d57dad912a35e81ee6c7"
server_key_version: "encryptionkey@0"
I20260806 15:58:11.311188 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:11.340241 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:11.341017 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:11.341234 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:11.341864 10177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:11.342830 10177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:11.342931 10177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:11.343047 10177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:11.343101 10177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:11.352211 10177 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:40817
I20260806 15:58:11.352253 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:40817 every 8 connection(s)
I20260806 15:58:11.353173 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:58:11.358927 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 10177
I20260806 15:58:11.359042 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:58:11.359360 14204 external_mini_cluster.cc:1428] Setting key 5eef4cd385fda7f64deb458df3fd6a31
I20260806 15:58:11.363036 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.354999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:33273 (local address 127.13.223.62:43537)
0806 15:58:11.355154 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.355157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.356286 (+ 1129us) server_negotiation.cc:407] Connection header received
0806 15:58:11.357810 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.357812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.357860 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.357942 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:11.358829 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.359595 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.361083 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.361298 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.361424 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.362626 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:11.362628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:11.362630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:11.362641 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:11.362658 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:11.362667 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:11.362723 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.362883 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.362932 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:58:11.363862 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.355398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:43537 (local address 127.13.223.3:33273)
0806 15:58:11.356119 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.356171 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.357578 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.358088 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.358095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.358182 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:11.358684 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.358696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.359787 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.359791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.360968 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.360980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.361506 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.361510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.361730 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.362379 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:11.362380 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:11.362419 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:11.362480 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:11.362741 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:11.362749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:11.362753 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.362865 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.363076 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
I20260806 15:58:11.364991 10367 heartbeater.cc:347] Connected to a master server at 127.13.223.62:43537
I20260806 15:58:11.365193 10367 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:11.365653 10367 heartbeater.cc:511] Master 127.13.223.62:43537 requested a full tablet report, sending...
I20260806 15:58:11.366614 9679 ts_manager.cc:194] Registered new tserver with Master: 91b129c472ab476993e1f82c29278007 (127.13.223.3:40817)
I20260806 15:58:11.367164 9679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.3:33273
I20260806 15:58:11.373569 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:58:11.379683 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.375716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.13.223.62:43537)
0806 15:58:11.375864 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.375880 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.375963 (+ 83us) server_negotiation.cc:407] Connection header received
0806 15:58:11.376101 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.376104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.376143 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.376199 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:11.376452 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.377020 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.378029 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.378240 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.378325 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.379031 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:11.379034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:11.379036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:11.379046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:11.379068 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:11.379093 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:11.379159 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.379359 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.379472 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 15:58:11.382671 9679 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:58:11.385816 9679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:58:11.399787 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.394755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.13.223.3:40817)
0806 15:58:11.395218 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.395230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.395256 (+ 26us) server_negotiation.cc:407] Connection header received
0806 15:58:11.395337 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.395342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.395554 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.395750 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:11.396140 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.396975 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.398496 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.399037 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.399185 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.399405 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.399524 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.399627 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":164,"threads_started":1}
I20260806 15:58:11.399781 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.394225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40817 (local address 127.0.0.1:37036)
0806 15:58:11.394963 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.394976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.395076 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.395739 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.395745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.395767 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:11.395999 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.396007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.397096 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.397099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.398369 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.398379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.399184 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.399188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.399328 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.399366 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.399484 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.399616 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":48,"thread_start_us":104,"threads_started":1}
I20260806 15:58:11.400058 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.394064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:40779 (local address 127.0.0.1:50840)
0806 15:58:11.394559 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.394617 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.394737 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.395822 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.395825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.395843 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:11.396034 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.396042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.397595 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.397598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.398578 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.398588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.399678 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.399681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.399768 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.399794 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.399844 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.399925 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":37,"thread_start_us":78,"threads_started":1}
I20260806 15:58:11.400931 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.394222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.13.223.2:40779)
0806 15:58:11.395331 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.395345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.395369 (+ 24us) server_negotiation.cc:407] Connection header received
0806 15:58:11.395444 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.395452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.395658 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.396169 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:11.396316 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.397474 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.398711 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.399517 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.400348 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.400561 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.400648 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.400748 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":112,"threads_started":1}
I20260806 15:58:11.401500 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.394728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36997 (local address 127.0.0.1:58292)
0806 15:58:11.395395 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.395407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.395487 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.395999 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.396002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.396019 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:11.396112 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.396119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.397595 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.397598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.399809 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.399818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.401102 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.401116 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.401211 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.401237 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.401313 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.401380 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":31,"thread_start_us":104,"threads_started":1}
I20260806 15:58:11.401945 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.395175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.13.223.1:36997)
0806 15:58:11.395596 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.395606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.395627 (+ 21us) server_negotiation.cc:407] Connection header received
0806 15:58:11.395699 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.395704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.395865 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.396007 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:11.396200 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.397456 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.399943 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.400752 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.400855 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.401036 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.401676 (+ 640us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.401783 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":139,"threads_started":1}
I20260806 15:58:11.403527 10281 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6624f76f5b4b32a1496e52b89362ec (DEFAULT_TABLE table=test-table [id=368d748d4b884e8cbc0a62e463098980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:11.403950 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6624f76f5b4b32a1496e52b89362ec (DEFAULT_TABLE table=test-table [id=368d748d4b884e8cbc0a62e463098980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:11.404191 9867 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6624f76f5b4b32a1496e52b89362ec (DEFAULT_TABLE table=test-table [id=368d748d4b884e8cbc0a62e463098980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:11.404754 10281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6624f76f5b4b32a1496e52b89362ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:11.404968 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6624f76f5b4b32a1496e52b89362ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:11.405206 9867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6624f76f5b4b32a1496e52b89362ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:11.410859 10394 tablet_bootstrap.cc:492] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: Bootstrap starting.
I20260806 15:58:11.412263 10395 tablet_bootstrap.cc:492] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Bootstrap starting.
I20260806 15:58:11.413275 10396 tablet_bootstrap.cc:492] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Bootstrap starting.
I20260806 15:58:11.413302 10394 tablet_bootstrap.cc:654] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:11.414469 10394 log.cc:824] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:11.414559 10395 tablet_bootstrap.cc:654] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:11.415349 10396 tablet_bootstrap.cc:654] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:11.415501 10395 log.cc:824] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:11.416324 10396 log.cc:824] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:11.417408 10394 tablet_bootstrap.cc:492] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: No bootstrap required, opened a new log
I20260806 15:58:11.417408 10395 tablet_bootstrap.cc:492] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: No bootstrap required, opened a new log
I20260806 15:58:11.417634 10395 ts_tablet_manager.cc:1397] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:58:11.417639 10394 ts_tablet_manager.cc:1397] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 15:58:11.418128 10396 tablet_bootstrap.cc:492] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: No bootstrap required, opened a new log
I20260806 15:58:11.418330 10396 ts_tablet_manager.cc:1397] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 15:58:11.421464 10396 raft_consensus.cc:348] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.421698 10396 raft_consensus.cc:374] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:11.421749 10396 raft_consensus.cc:729] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb02370972d4adfa4f375d7be29902f, State: Initialized, Role: FOLLOWER
I20260806 15:58:11.421936 10394 raft_consensus.cc:348] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.421936 10395 raft_consensus.cc:348] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.422173 10395 raft_consensus.cc:374] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:11.422173 10394 raft_consensus.cc:374] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:11.422231 10394 raft_consensus.cc:729] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bcbc6ba5ee4d0d819e5e468dc00c1d, State: Initialized, Role: FOLLOWER
I20260806 15:58:11.422214 10396 consensus_queue.cc:260] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.422231 10395 raft_consensus.cc:729] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b129c472ab476993e1f82c29278007, State: Initialized, Role: FOLLOWER
I20260806 15:58:11.422694 10394 consensus_queue.cc:260] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.422925 10395 consensus_queue.cc:260] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.423223 10396 ts_tablet_manager.cc:1428] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 15:58:11.423534 10394 ts_tablet_manager.cc:1428] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:58:11.423565 10367 heartbeater.cc:503] Master 127.13.223.62:43537 was elected leader, sending a full tablet report...
I20260806 15:58:11.423565 10173 heartbeater.cc:503] Master 127.13.223.62:43537 was elected leader, sending a full tablet report...
I20260806 15:58:11.423803 10395 ts_tablet_manager.cc:1428] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 15:58:11.423858 9980 heartbeater.cc:503] Master 127.13.223.62:43537 was elected leader, sending a full tablet report...
W20260806 15:58:11.454589 9981 tablet.cc:2369] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:58:11.604171 10368 tablet.cc:2369] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:58:11.638962 10402 raft_consensus.cc:482] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:11.639206 10402 raft_consensus.cc:504] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.640411 10402 leader_election.cc:288] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91b129c472ab476993e1f82c29278007 (127.13.223.3:40817), a4bcbc6ba5ee4d0d819e5e468dc00c1d (127.13.223.1:36997)
I20260806 15:58:11.644562 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.640996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:52543 (local address 127.13.223.1:36997)
0806 15:58:11.641141 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.641146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.641324 (+ 178us) server_negotiation.cc:407] Connection header received
0806 15:58:11.641485 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.641488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.641559 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.641645 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:11.641924 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.642399 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.643542 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.643967 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.644102 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.644153 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.644347 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.644414 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 15:58:11.644578 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.640748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36997 (local address 127.13.223.2:52543)
0806 15:58:11.641234 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.641245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.641354 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.641657 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.641659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.641675 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:11.641813 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.641818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.642510 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.642515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.643402 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.643411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.644107 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.644110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.644216 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.644241 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.644329 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.644398 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":119,"thread_start_us":135,"threads_started":1}
I20260806 15:58:11.644856 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.640748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:40817 (local address 127.13.223.2:45401)
0806 15:58:11.640865 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.640883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:11.640984 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:11.641402 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:11.641406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:11.641428 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:11.641643 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.641652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.642493 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.642508 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.643463 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:11.643474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.644298 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.644302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:11.644429 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.644450 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 15:58:11.644617 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 15:58:11.644702 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":47}
I20260806 15:58:11.644856 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.640938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:45401 (local address 127.13.223.3:40817)
0806 15:58:11.641099 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.641104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.641126 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:58:11.641207 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.641210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.641256 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.641385 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:11.641795 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.642355 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.643643 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.644144 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.644305 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.644363 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.644638 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.644719 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 15:58:11.645366 9899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6624f76f5b4b32a1496e52b89362ec" candidate_uuid: "eeb02370972d4adfa4f375d7be29902f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" is_pre_election: true
I20260806 15:58:11.645546 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6624f76f5b4b32a1496e52b89362ec" candidate_uuid: "eeb02370972d4adfa4f375d7be29902f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b129c472ab476993e1f82c29278007" is_pre_election: true
I20260806 15:58:11.645735 9899 raft_consensus.cc:2454] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb02370972d4adfa4f375d7be29902f in term 0.
I20260806 15:58:11.645826 10305 raft_consensus.cc:2454] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb02370972d4adfa4f375d7be29902f in term 0.
I20260806 15:58:11.646382 10006 leader_election.cc:302] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bcbc6ba5ee4d0d819e5e468dc00c1d, eeb02370972d4adfa4f375d7be29902f; no voters:
I20260806 15:58:11.646677 10402 raft_consensus.cc:2781] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:58:11.646777 10402 raft_consensus.cc:482] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:58:11.646835 10402 raft_consensus.cc:3037] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:11.647919 10402 raft_consensus.cc:504] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.648283 10402 leader_election.cc:288] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [CANDIDATE]: Term 1 election: Requested vote from peers 91b129c472ab476993e1f82c29278007 (127.13.223.3:40817), a4bcbc6ba5ee4d0d819e5e468dc00c1d (127.13.223.1:36997)
I20260806 15:58:11.648679 9899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6624f76f5b4b32a1496e52b89362ec" candidate_uuid: "eeb02370972d4adfa4f375d7be29902f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d"
I20260806 15:58:11.648676 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6624f76f5b4b32a1496e52b89362ec" candidate_uuid: "eeb02370972d4adfa4f375d7be29902f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b129c472ab476993e1f82c29278007"
I20260806 15:58:11.648790 9899 raft_consensus.cc:3037] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:11.648824 10305 raft_consensus.cc:3037] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:11.649855 10305 raft_consensus.cc:2454] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb02370972d4adfa4f375d7be29902f in term 1.
I20260806 15:58:11.649883 9899 raft_consensus.cc:2454] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb02370972d4adfa4f375d7be29902f in term 1.
I20260806 15:58:11.650162 10007 leader_election.cc:302] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b129c472ab476993e1f82c29278007, eeb02370972d4adfa4f375d7be29902f; no voters:
I20260806 15:58:11.650337 10402 raft_consensus.cc:2781] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:11.650552 10402 raft_consensus.cc:686] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 1 LEADER]: Becoming Leader. State: Replica: eeb02370972d4adfa4f375d7be29902f, State: Running, Role: LEADER
I20260806 15:58:11.650894 10402 consensus_queue.cc:237] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } }
I20260806 15:58:11.654055 9679 catalog_manager.cc:5495] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb02370972d4adfa4f375d7be29902f (127.13.223.2). New cstate: current_term: 1 leader_uuid: "eeb02370972d4adfa4f375d7be29902f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb02370972d4adfa4f375d7be29902f" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 40779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 } health_report { overall_health: UNKNOWN } } }
W20260806 15:58:11.655318 10174 tablet.cc:2369] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:58:11.691618 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.688030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.13.223.2:40779)
0806 15:58:11.688196 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:11.688202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:11.688219 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:58:11.688306 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:11.688309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:11.688364 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:11.688474 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:11.688753 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.689244 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.690138 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:11.690366 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:11.690479 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:11.690785 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:11.691195 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:11.691328 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:11.691389 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 15:58:11.691452 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 15:58:11.697638 10305 raft_consensus.cc:1261] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 1 FOLLOWER]: Refusing update from remote peer eeb02370972d4adfa4f375d7be29902f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:11.697646 9899 raft_consensus.cc:1261] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d [term 1 FOLLOWER]: Refusing update from remote peer eeb02370972d4adfa4f375d7be29902f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:11.698364 10406 consensus_queue.cc:1035] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bcbc6ba5ee4d0d819e5e468dc00c1d" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:11.698554 10402 consensus_queue.cc:1035] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b129c472ab476993e1f82c29278007" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:11.705456 10410 mvcc.cc:205] Tried to move back new op lower bound from 7315586628385566720 to 7315586628204793856. Current Snapshot: MvccSnapshot[applied={T|T < 7315586628385566720}]
I20260806 15:58:11.726866 9679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:11.727046 9679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:11.729537 9679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd17b6ad39104fae934922247c31f097: Sending DeleteTablet for 3 replicas of tablet 9c6624f76f5b4b32a1496e52b89362ec
I20260806 15:58:11.730316 10075 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6624f76f5b4b32a1496e52b89362ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:11 UTC) from {username='slave'} at 127.0.0.1:50840
I20260806 15:58:11.730429 9867 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6624f76f5b4b32a1496e52b89362ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:11 UTC) from {username='slave'} at 127.0.0.1:58292
I20260806 15:58:11.730866 10281 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6624f76f5b4b32a1496e52b89362ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:11 UTC) from {username='slave'} at 127.0.0.1:37036
I20260806 15:58:11.731154 10429 tablet_replica.cc:330] T 9c6624f76f5b4b32a1496e52b89362ec P a4bcbc6ba5ee4d0d819e5e468dc00c1d: stopping tablet replica
I20260806 15:58:11.731199 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 9773
I20260806 15:58:11.731972 10431 tablet_replica.cc:330] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: stopping tablet replica
I20260806 15:58:11.734375 10431 raft_consensus.cc:2229] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:11.734866 10431 raft_consensus.cc:2258] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:11.739053 10431 ts_tablet_manager.cc:1905] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:11.742238 10431 ts_tablet_manager.cc:1918] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:11.742332 10431 log.cc:1196] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/9c6624f76f5b4b32a1496e52b89362ec
I20260806 15:58:11.742627 10431 ts_tablet_manager.cc:1939] T 9c6624f76f5b4b32a1496e52b89362ec P 91b129c472ab476993e1f82c29278007: Deleting consensus metadata
I20260806 15:58:11.743978 9658 catalog_manager.cc:4841] TS 91b129c472ab476993e1f82c29278007 (127.13.223.3:40817): tablet 9c6624f76f5b4b32a1496e52b89362ec (table test-table [id=368d748d4b884e8cbc0a62e463098980]) successfully deleted
I20260806 15:58:11.747082 10428 tablet_replica.cc:330] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: stopping tablet replica
I20260806 15:58:11.747488 10428 raft_consensus.cc:2229] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:58:11.747905 10428 raft_consensus.cc:2258] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:11.749577 10428 ts_tablet_manager.cc:1905] T 9c6624f76f5b4b32a1496e52b89362ec P eeb02370972d4adfa4f375d7be29902f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 15:58:11.751120 9657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:36997 (error 108)
I20260806 15:58:11.751377 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 9983
I20260806 15:58:11.760068 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.759737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36997 (local address 127.0.0.1:58308)
0806 15:58:11.759911 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.759986 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 15:58:11.760389 9657 catalog_manager.cc:4584] TS a4bcbc6ba5ee4d0d819e5e468dc00c1d (127.13.223.1:36997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c6624f76f5b4b32a1496e52b89362ec: Network error: Client connection negotiation failed: client connection to 127.13.223.1:36997: connect: Connection refused (error 111)
I20260806 15:58:11.767478 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 10177
I20260806 15:58:11.771806 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:11.767666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:40779 (local address 127.0.0.1:50860)
0806 15:58:11.771666 (+ 4000us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:11.771723 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.2:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3902}
I20260806 15:58:11.777765 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 9624
2026-08-06T15:58:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1282 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:11 dist-test-slave-rwrg krb5kdc[10437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:11 dist-test-slave-rwrg krb5kdc[10437](info): set up 2 sockets
Aug 06 15:58:11 dist-test-slave-rwrg krb5kdc[10437](info): commencing operation
krb5kdc: starting...
W20260806 15:58:13.145462 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:13.275739 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:46765
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:13.390196 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:13.390544 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:13.390648 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:13.394727 10453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:13.394809 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:13.394858 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:13.394909 10453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:13.394976 10453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:13.400475 10453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:13.401675 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:13.402776 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:13.411518 10461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:13.411507 10464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:13.411509 10462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:13.412240 10453 server_base.cc:1034] running on GCE node
I20260806 15:58:13.412947 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:13.414109 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:13.415306 10453 hybrid_clock.cc:648] HybridClock initialized: now 1786031893415292 us; error 27 us; skew 500 ppm
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:13.423138 10453 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:13.424090 10453 webserver.cc:460] Webserver started at http://127.13.223.62:39679/ using document root <none> and password file <none>
I20260806 15:58:13.424759 10453 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:13.424854 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:13.425076 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:13.427148 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "2a332b82e69340be853eb9619df26aea"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "5fae153383f1f9a670a1175cd6a5b267"
server_key_iv: "6e8cf8d4d1ccb349c43d9d9487adb194"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.427770 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "2a332b82e69340be853eb9619df26aea"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "5fae153383f1f9a670a1175cd6a5b267"
server_key_iv: "6e8cf8d4d1ccb349c43d9d9487adb194"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.431648 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:58:13.434310 10476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.435750 10453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 15:58:13.435886 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "2a332b82e69340be853eb9619df26aea"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "5fae153383f1f9a670a1175cd6a5b267"
server_key_iv: "6e8cf8d4d1ccb349c43d9d9487adb194"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.435995 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:13.466841 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:13.470379 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:13.470654 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:13.478121 10453 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:46765
I20260806 15:58:13.478178 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:46765 every 8 connection(s)
I20260806 15:58:13.479138 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:13.482075 10567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:13.483269 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 10453
I20260806 15:58:13.483426 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:13.483749 14204 external_mini_cluster.cc:1428] Setting key 75843f19a9dbd38c5a8b3d76fc8f984d
I20260806 15:58:13.489347 10567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea: Bootstrap starting.
I20260806 15:58:13.492095 10567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:13.493032 10567 log.cc:824] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea: Log is configured to *not* fsync() on all Append() calls
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:13.495364 10567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea: No bootstrap required, opened a new log
I20260806 15:58:13.499166 10567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a332b82e69340be853eb9619df26aea" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46765 } }
I20260806 15:58:13.499431 10567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:13.499497 10567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a332b82e69340be853eb9619df26aea, State: Initialized, Role: FOLLOWER
I20260806 15:58:13.499981 10567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a332b82e69340be853eb9619df26aea" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46765 } }
I20260806 15:58:13.500110 10567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:13.500183 10567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:13.500301 10567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:13.500602 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:13.485376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.13.223.62:46765)
0806 15:58:13.485826 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:13.485844 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:13.485883 (+ 39us) server_negotiation.cc:407] Connection header received
0806 15:58:13.486574 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:13.486593 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:13.486977 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:13.487300 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:13.488391 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.490121 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.491433 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.491994 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.492197 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:13.495329 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:13.495335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:13.495350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:13.495381 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:13.497375 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:13.497852 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:13.497863 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:13.497868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:13.497983 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:13.498279 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:13.498284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:13.498285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:13.498706 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:13.498929 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:13.499234 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 15:58:13.499580 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260806 15:58:13.501504 10567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a332b82e69340be853eb9619df26aea" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46765 } }
I20260806 15:58:13.501879 10567 leader_election.cc:302] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a332b82e69340be853eb9619df26aea; no voters:
I20260806 15:58:13.502223 10567 leader_election.cc:288] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:13.502341 10575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:13.502514 10575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [term 1 LEADER]: Becoming Leader. State: Replica: 2a332b82e69340be853eb9619df26aea, State: Running, Role: LEADER
I20260806 15:58:13.502803 10575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a332b82e69340be853eb9619df26aea" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46765 } }
I20260806 15:58:13.503561 10567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:13.504046 10577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a332b82e69340be853eb9619df26aea. Latest consensus state: current_term: 1 leader_uuid: "2a332b82e69340be853eb9619df26aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a332b82e69340be853eb9619df26aea" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46765 } } }
I20260806 15:58:13.504218 10577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:13.504514 10576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a332b82e69340be853eb9619df26aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a332b82e69340be853eb9619df26aea" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 46765 } } }
I20260806 15:58:13.504696 10576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:13.506165 10582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:13.509471 10582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:13.515018 10582 catalog_manager.cc:1347] Generated new cluster ID: 29697337eb4a4c428fb46b93f96b7974
I20260806 15:58:13.515089 10582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:13.524319 10582 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:13.525264 10582 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:13.529806 10582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea: Generated new TSK 0
I20260806 15:58:13.530658 10582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:58:13.569244 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:46765
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:13.683696 10601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:13.683977 10601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:13.684048 10601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:13.688091 10601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:13.688175 10601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:13.688282 10601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:58:13.693449 10601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:13.694506 10601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:13.695564 10601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:13.705240 10608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:13.705256 10607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:13.705562 10601 server_base.cc:1034] running on GCE node
W20260806 15:58:13.705462 10610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:13.706387 10601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:13.707429 10601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:13.711344 10601 hybrid_clock.cc:648] HybridClock initialized: now 1786031893711311 us; error 41 us; skew 500 ppm
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:13.714298 10601 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:58:13.715181 10601 webserver.cc:460] Webserver started at http://127.13.223.1:40263/ using document root <none> and password file <none>
I20260806 15:58:13.715701 10601 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:13.715758 10601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:13.715978 10601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:13.718113 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "47ae3d43724942dea3a837bbcb61601b"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "7a64f389af7dc91e97dd77d63cbf06b8"
server_key_iv: "590ad0c794f3eac52e7cab77c8ac9d38"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.718685 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "47ae3d43724942dea3a837bbcb61601b"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "7a64f389af7dc91e97dd77d63cbf06b8"
server_key_iv: "590ad0c794f3eac52e7cab77c8ac9d38"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.722748 10601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:13.725268 10624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.726456 10601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:13.726593 10601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "47ae3d43724942dea3a837bbcb61601b"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "7a64f389af7dc91e97dd77d63cbf06b8"
server_key_iv: "590ad0c794f3eac52e7cab77c8ac9d38"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.726702 10601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:13.748615 10601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:13.752136 10601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:13.752345 10601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:13.752996 10601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:13.753902 10601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:13.753966 10601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.754032 10601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:13.754081 10601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.763657 10601 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:36125
I20260806 15:58:13.763710 10806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:36125 every 8 connection(s)
I20260806 15:58:13.764587 10601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:58:13.766795 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 10601
I20260806 15:58:13.766943 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:58:13.767249 14204 external_mini_cluster.cc:1428] Setting key 504ed9a38557e334bdf75dfc16952c92
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
I20260806 15:58:13.779989 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:13.766462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:59767 (local address 127.13.223.62:46765)
0806 15:58:13.766678 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:13.766683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:13.767850 (+ 1167us) server_negotiation.cc:407] Connection header received
0806 15:58:13.768914 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:13.768918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:13.768969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:13.769051 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:13.770223 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.771069 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.772898 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.773126 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.773209 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:13.776260 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:13.776264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:13.776266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:13.776296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:13.777770 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:13.778349 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:13.778357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:13.778358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:13.778429 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:13.778795 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:13.778798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:13.778799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:13.779034 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:13.779138 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:13.779635 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 15:58:13.779765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
I20260806 15:58:13.780966 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:13.766881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:46765 (local address 127.13.223.1:59767)
0806 15:58:13.767691 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:13.767748 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:13.768726 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:13.769188 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:13.769195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:13.769563 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:13.770065 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:13.770079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.771204 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.771210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:13.772765 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:13.772781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.773287 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.773294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:13.773526 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:13.774265 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:13.774292 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:13.776074 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:13.777893 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:13.777899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:13.777914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:13.778242 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:13.778516 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:13.778519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:13.778521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:13.778669 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:13.779137 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:13.779145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:13.779391 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 15:58:13.779582 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 15:58:13.779901 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":176,"threads_started":1}
I20260806 15:58:13.782317 10807 heartbeater.cc:347] Connected to a master server at 127.13.223.62:46765
I20260806 15:58:13.782598 10807 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:13.783123 10807 heartbeater.cc:511] Master 127.13.223.62:46765 requested a full tablet report, sending...
I20260806 15:58:13.784853 10506 ts_manager.cc:194] Registered new tserver with Master: 47ae3d43724942dea3a837bbcb61601b (127.13.223.1:36125)
I20260806 15:58:13.786620 10506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:59767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:58:13.799850 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:46765
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:13.913055 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:13.913326 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:13.913368 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:13.917409 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:13.917461 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:13.917517 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:58:13.922729 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:13.923840 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:13.924860 10816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:13.934912 10827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:13.934916 10829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:13.935063 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:13.935297 10816 server_base.cc:1034] running on GCE node
I20260806 15:58:13.935710 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:13.936362 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:13.937536 10816 hybrid_clock.cc:648] HybridClock initialized: now 1786031893937516 us; error 30 us; skew 500 ppm
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:13.940450 10816 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:58:13.941345 10816 webserver.cc:460] Webserver started at http://127.13.223.2:33405/ using document root <none> and password file <none>
I20260806 15:58:13.941946 10816 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:13.942010 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:13.942176 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:13.944224 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "6caf99b397b6496da912b940275b8e12"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "f1f8e9748437ce0b94c3625dc47bbc0d"
server_key_iv: "2e1fe9478faed317b0f1741d1f8c5aa9"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.944808 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "6caf99b397b6496da912b940275b8e12"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "f1f8e9748437ce0b94c3625dc47bbc0d"
server_key_iv: "2e1fe9478faed317b0f1741d1f8c5aa9"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.948429 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:13.950940 10843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.952273 10816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 15:58:13.952402 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "6caf99b397b6496da912b940275b8e12"
format_stamp: "Formatted at 2026-08-06 15:58:13 on dist-test-slave-rwrg"
server_key: "f1f8e9748437ce0b94c3625dc47bbc0d"
server_key_iv: "2e1fe9478faed317b0f1741d1f8c5aa9"
server_key_version: "encryptionkey@0"
I20260806 15:58:13.952597 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:13.971168 10816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:13.974464 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:13.974679 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:13.975284 10816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:13.976128 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:13.976176 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.976244 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:13.976280 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:13.985098 10816 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:45689
I20260806 15:58:13.985184 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:45689 every 8 connection(s)
I20260806 15:58:13.986044 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:58:13.987424 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 10816
I20260806 15:58:13.987571 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:58:13.987903 14204 external_mini_cluster.cc:1428] Setting key dbd2c35eae1de421bee94877ee519627
Aug 06 15:58:13 dist-test-slave-rwrg krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:58:14.001016 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:13.988025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:56265 (local address 127.13.223.62:46765)
0806 15:58:13.988194 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:13.988199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:13.989569 (+ 1370us) server_negotiation.cc:407] Connection header received
0806 15:58:13.990555 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:13.990560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:13.990624 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:13.990718 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:13.991964 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.992640 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.993877 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.994139 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.994256 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:13.997008 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:13.997013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:13.997016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:13.997043 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:13.998774 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:13.999299 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:13.999301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:13.999302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:13.999398 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:13.999820 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:13.999824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:13.999825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:14.000025 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:14.000145 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.000760 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.000885 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:58:14.001919 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:13.988417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:46765 (local address 127.13.223.2:56265)
0806 15:58:13.989363 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:13.989428 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:13.990346 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:13.990871 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:13.990878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:13.991264 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:13.991814 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:13.991828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.992781 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.992790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:13.993745 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:13.993755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:13.994300 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:13.994304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:13.994474 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:13.995112 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:13.995129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:13.996796 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:13.998910 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:13.998914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:13.998926 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:13.999193 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:13.999518 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:13.999524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:13.999529 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:13.999712 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:14.000143 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:14.000151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:14.000398 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.000676 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.000981 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":174,"threads_started":1}
I20260806 15:58:14.003201 11023 heartbeater.cc:347] Connected to a master server at 127.13.223.62:46765
I20260806 15:58:14.003500 11023 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:14.004087 11023 heartbeater.cc:511] Master 127.13.223.62:46765 requested a full tablet report, sending...
I20260806 15:58:14.005425 10506 ts_manager.cc:194] Registered new tserver with Master: 6caf99b397b6496da912b940275b8e12 (127.13.223.2:45689)
I20260806 15:58:14.006119 10506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:56265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:58:14.019502 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:46765
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:14.133466 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:14.133718 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:14.133790 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:14.138021 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:14.138087 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:14.138185 11029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:58:14.143738 11029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:14.144903 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:14.146051 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:14.155821 11039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:14.155869 11040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:14.155900 11043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:14.156246 11029 server_base.cc:1034] running on GCE node
I20260806 15:58:14.156733 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:14.157435 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:14.158622 11029 hybrid_clock.cc:648] HybridClock initialized: now 1786031894158605 us; error 36 us; skew 500 ppm
Aug 06 15:58:14 dist-test-slave-rwrg krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:14.161489 11029 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:58:14.162349 11029 webserver.cc:460] Webserver started at http://127.13.223.3:33831/ using document root <none> and password file <none>
I20260806 15:58:14.162925 11029 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:14.163021 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:14.163262 11029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:14.165354 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "97b48ba7a6e840a8bce668ca25bfb105"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "9cc318bc4037e49bc11c2350463bf5cc"
server_key_iv: "21a6753c2091c3d71b80023802708bfe"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.165916 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "97b48ba7a6e840a8bce668ca25bfb105"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "9cc318bc4037e49bc11c2350463bf5cc"
server_key_iv: "21a6753c2091c3d71b80023802708bfe"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.169608 11029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 15:58:14.171878 11057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.173092 11029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:14.173193 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "97b48ba7a6e840a8bce668ca25bfb105"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "9cc318bc4037e49bc11c2350463bf5cc"
server_key_iv: "21a6753c2091c3d71b80023802708bfe"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.173272 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:14.193167 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:14.196062 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:14.196223 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:14.196900 11029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:14.197682 11029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:14.197731 11029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.197770 11029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:14.197789 11029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.207161 11029 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:37147
I20260806 15:58:14.207190 11229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:37147 every 8 connection(s)
I20260806 15:58:14.208043 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:58:14.216943 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 11029
I20260806 15:58:14.217084 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
Aug 06 15:58:14 dist-test-slave-rwrg krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:14.217360 14204 external_mini_cluster.cc:1428] Setting key b6e932966a1dceb1eb36097a6c11dfe6
I20260806 15:58:14.221966 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.209900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:39937 (local address 127.13.223.62:46765)
0806 15:58:14.210032 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.210037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.211094 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 15:58:14.211980 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.211983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.212043 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.212124 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:14.213366 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.213902 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.215027 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.215253 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.215364 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.218050 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:14.218059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:14.218062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:14.218087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:14.219825 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:14.220363 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:14.220366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:14.220367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:14.220444 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:14.220958 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:14.220961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:14.220962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:14.221149 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:14.221229 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.221748 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.221849 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:58:14.222892 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.210279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:46765 (local address 127.13.223.3:39937)
0806 15:58:14.210950 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.210992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.211779 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.212292 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.212298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.212721 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:14.213215 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.213229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.214030 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.214034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.214899 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.214908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.215405 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.215410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.215577 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.216134 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:14.216150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:14.217903 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:14.219949 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:14.219954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:14.219967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:14.220273 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:14.220695 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:14.220698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:14.220700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:14.220862 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:14.221245 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:14.221251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:14.221522 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.221705 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.221976 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":107,"threads_started":1}
I20260806 15:58:14.224041 11232 heartbeater.cc:347] Connected to a master server at 127.13.223.62:46765
I20260806 15:58:14.224310 11232 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:14.224927 11232 heartbeater.cc:511] Master 127.13.223.62:46765 requested a full tablet report, sending...
I20260806 15:58:14.226004 10506 ts_manager.cc:194] Registered new tserver with Master: 97b48ba7a6e840a8bce668ca25bfb105 (127.13.223.3:37147)
I20260806 15:58:14.226583 10506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:39937
I20260806 15:58:14.231498 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:58:14.241433 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.233560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.13.223.62:46765)
0806 15:58:14.233689 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.233695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.233885 (+ 190us) server_negotiation.cc:407] Connection header received
0806 15:58:14.234031 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.234035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.234089 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.234170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:14.234873 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.235385 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.236207 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.236435 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.236569 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.238215 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:14.238219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:14.238222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:14.238248 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:14.239556 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:14.240074 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:14.240077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:14.240079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:14.240168 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:14.240535 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:14.240539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:14.240540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:14.240744 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:14.240904 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.241181 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.241274 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 15:58:14.244059 10506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:58:14.247025 10506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:58:14.262167 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.255793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.13.223.1:36125)
0806 15:58:14.257437 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.257458 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.257488 (+ 30us) server_negotiation.cc:407] Connection header received
0806 15:58:14.257577 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.257582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.257807 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.258042 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:14.258310 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.259099 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.260867 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.261469 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.261600 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.261755 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.261911 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.262007 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":85,"threads_started":1}
I20260806 15:58:14.262157 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.255640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36125 (local address 127.0.0.1:40816)
0806 15:58:14.256532 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.256588 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.256721 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.257998 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.258001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.258029 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:14.258166 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.258175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.259242 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.259249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.260718 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.260731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.261600 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.261603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.261710 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.261746 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.261881 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.262007 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":1,"spinlock_wait_cycles":2688,"thread_start_us":83,"threads_started":1}
I20260806 15:58:14.262704 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.256020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37147 (local address 127.0.0.1:48492)
0806 15:58:14.256920 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.256930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.256986 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.257755 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.257783 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.257811 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:14.258209 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.258216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.259627 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.259630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.260812 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.260826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.262186 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.262190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.262338 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.262378 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.262457 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.262543 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":111,"thread_start_us":94,"threads_started":1}
I20260806 15:58:14.263180 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.256427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.13.223.2:45689)
0806 15:58:14.257269 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.257279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.257302 (+ 23us) server_negotiation.cc:407] Connection header received
0806 15:58:14.257373 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.257378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.257551 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.258011 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:14.258207 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.259406 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.260988 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.261835 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.261957 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.262109 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.262931 (+ 822us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.263035 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":127,"threads_started":1}
I20260806 15:58:14.263219 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.256634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.13.223.3:37147)
0806 15:58:14.256987 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.256999 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.257023 (+ 24us) server_negotiation.cc:407] Connection header received
0806 15:58:14.257127 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.257134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.257342 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.257492 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:14.258342 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.259466 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.260959 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.261835 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.261957 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.262109 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.262959 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.263069 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":166,"threads_started":1}
I20260806 15:58:14.263208 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.255760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:45689 (local address 127.0.0.1:39706)
0806 15:58:14.256532 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.256588 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.256721 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.257795 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.257798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.257816 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:14.258030 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.258038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.259565 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.259569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.260839 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.260850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.262186 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.262190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.262674 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.262699 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.262797 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.263070 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":450,"spinlock_wait_cycles":11392,"thread_start_us":85,"threads_started":1}
I20260806 15:58:14.266451 11128 tablet_service.cc:1467] Processing CreateTablet for tablet 132a66eb862d48f9b28a908c3e4e1c0f (DEFAULT_TABLE table=test-table [id=091bf72f541745aca4578492ccbe955d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:14.266451 10703 tablet_service.cc:1467] Processing CreateTablet for tablet 132a66eb862d48f9b28a908c3e4e1c0f (DEFAULT_TABLE table=test-table [id=091bf72f541745aca4578492ccbe955d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:14.266631 10913 tablet_service.cc:1467] Processing CreateTablet for tablet 132a66eb862d48f9b28a908c3e4e1c0f (DEFAULT_TABLE table=test-table [id=091bf72f541745aca4578492ccbe955d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:14.267520 10913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132a66eb862d48f9b28a908c3e4e1c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:14.267551 10703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132a66eb862d48f9b28a908c3e4e1c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:14.267568 11128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132a66eb862d48f9b28a908c3e4e1c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:14.274034 11260 tablet_bootstrap.cc:492] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Bootstrap starting.
I20260806 15:58:14.276780 11260 tablet_bootstrap.cc:654] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:14.277102 11264 tablet_bootstrap.cc:492] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Bootstrap starting.
I20260806 15:58:14.277758 11260 log.cc:824] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:14.278466 11261 tablet_bootstrap.cc:492] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: Bootstrap starting.
I20260806 15:58:14.279685 11260 tablet_bootstrap.cc:492] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: No bootstrap required, opened a new log
I20260806 15:58:14.279896 11260 ts_tablet_manager.cc:1397] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 15:58:14.279961 11264 tablet_bootstrap.cc:654] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:14.280901 11261 tablet_bootstrap.cc:654] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:14.280902 11264 log.cc:824] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:14.282258 11261 log.cc:824] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:14.283198 11264 tablet_bootstrap.cc:492] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: No bootstrap required, opened a new log
I20260806 15:58:14.283399 11264 ts_tablet_manager.cc:1397] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:58:14.283475 11260 raft_consensus.cc:348] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.283751 11260 raft_consensus.cc:374] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:14.283816 11260 raft_consensus.cc:729] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caf99b397b6496da912b940275b8e12, State: Initialized, Role: FOLLOWER
I20260806 15:58:14.284248 11260 consensus_queue.cc:260] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.284337 11261 tablet_bootstrap.cc:492] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: No bootstrap required, opened a new log
I20260806 15:58:14.284557 11261 ts_tablet_manager.cc:1397] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:58:14.284991 11023 heartbeater.cc:503] Master 127.13.223.62:46765 was elected leader, sending a full tablet report...
I20260806 15:58:14.285306 11260 ts_tablet_manager.cc:1428] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260806 15:58:14.287696 11264 raft_consensus.cc:348] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.288038 11264 raft_consensus.cc:374] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:14.288108 11264 raft_consensus.cc:729] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b48ba7a6e840a8bce668ca25bfb105, State: Initialized, Role: FOLLOWER
I20260806 15:58:14.288623 11264 consensus_queue.cc:260] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.288702 11261 raft_consensus.cc:348] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.288971 11261 raft_consensus.cc:374] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:14.289041 11261 raft_consensus.cc:729] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ae3d43724942dea3a837bbcb61601b, State: Initialized, Role: FOLLOWER
I20260806 15:58:14.289477 11261 consensus_queue.cc:260] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.289716 11264 ts_tablet_manager.cc:1428] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:58:14.290092 11232 heartbeater.cc:503] Master 127.13.223.62:46765 was elected leader, sending a full tablet report...
I20260806 15:58:14.290465 11261 ts_tablet_manager.cc:1428] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 15:58:14.290838 10807 heartbeater.cc:503] Master 127.13.223.62:46765 was elected leader, sending a full tablet report...
I20260806 15:58:14.295825 11268 raft_consensus.cc:482] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:14.295989 11268 raft_consensus.cc:504] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.297161 11268 leader_election.cc:288] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47ae3d43724942dea3a837bbcb61601b (127.13.223.1:36125), 97b48ba7a6e840a8bce668ca25bfb105 (127.13.223.3:37147)
I20260806 15:58:14.301237 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.297373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36125 (local address 127.13.223.2:48345)
0806 15:58:14.297510 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.297563 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.297652 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.297991 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.297994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.298014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:14.298158 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.298165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.299127 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.299130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.300011 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.300020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.300823 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.300827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.300935 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.300961 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.301039 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.301108 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260806 15:58:14.301261 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.297615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:48345 (local address 127.13.223.1:36125)
0806 15:58:14.297749 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.297756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.297770 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:14.297816 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.297819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.297865 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.297972 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:14.298282 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.299025 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.300131 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.300714 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.300806 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.300845 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.301067 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.301136 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:58:14.301517 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.297469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:45551 (local address 127.13.223.3:37147)
0806 15:58:14.297587 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.297595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.297975 (+ 380us) server_negotiation.cc:407] Connection header received
0806 15:58:14.298178 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.298182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.298236 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.298318 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:14.298588 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.299274 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.300396 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.300940 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.301062 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.301118 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.301309 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.301383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 15:58:14.301527 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.297409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:37147 (local address 127.13.223.2:45551)
0806 15:58:14.297894 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.297909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.298061 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.298326 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.298329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.298343 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:14.298472 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.298477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.299390 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.299393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.300279 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.300289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.301079 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.301083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.301177 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.301199 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.301281 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.301352 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":47,"spinlock_wait_cycles":22912,"thread_start_us":146,"threads_started":1}
I20260806 15:58:14.302119 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132a66eb862d48f9b28a908c3e4e1c0f" candidate_uuid: "6caf99b397b6496da912b940275b8e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b48ba7a6e840a8bce668ca25bfb105" is_pre_election: true
I20260806 15:58:14.302196 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132a66eb862d48f9b28a908c3e4e1c0f" candidate_uuid: "6caf99b397b6496da912b940275b8e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ae3d43724942dea3a837bbcb61601b" is_pre_election: true
I20260806 15:58:14.302419 11164 raft_consensus.cc:2454] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caf99b397b6496da912b940275b8e12 in term 0.
I20260806 15:58:14.302451 10732 raft_consensus.cc:2454] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caf99b397b6496da912b940275b8e12 in term 0.
I20260806 15:58:14.302814 10845 leader_election.cc:302] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caf99b397b6496da912b940275b8e12, 97b48ba7a6e840a8bce668ca25bfb105; no voters:
I20260806 15:58:14.303020 11268 raft_consensus.cc:2781] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:58:14.303128 11268 raft_consensus.cc:482] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:58:14.303189 11268 raft_consensus.cc:3037] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:14.304339 11268 raft_consensus.cc:504] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.304734 11268 leader_election.cc:288] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [CANDIDATE]: Term 1 election: Requested vote from peers 47ae3d43724942dea3a837bbcb61601b (127.13.223.1:36125), 97b48ba7a6e840a8bce668ca25bfb105 (127.13.223.3:37147)
I20260806 15:58:14.305070 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132a66eb862d48f9b28a908c3e4e1c0f" candidate_uuid: "6caf99b397b6496da912b940275b8e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ae3d43724942dea3a837bbcb61601b"
I20260806 15:58:14.305132 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132a66eb862d48f9b28a908c3e4e1c0f" candidate_uuid: "6caf99b397b6496da912b940275b8e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b48ba7a6e840a8bce668ca25bfb105"
I20260806 15:58:14.305202 10732 raft_consensus.cc:3037] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:14.305294 11164 raft_consensus.cc:3037] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:14.306269 10732 raft_consensus.cc:2454] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6caf99b397b6496da912b940275b8e12 in term 1.
I20260806 15:58:14.306391 11164 raft_consensus.cc:2454] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6caf99b397b6496da912b940275b8e12 in term 1.
I20260806 15:58:14.306671 10844 leader_election.cc:302] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ae3d43724942dea3a837bbcb61601b, 6caf99b397b6496da912b940275b8e12; no voters:
I20260806 15:58:14.306871 11268 raft_consensus.cc:2781] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:14.307087 11268 raft_consensus.cc:686] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 1 LEADER]: Becoming Leader. State: Replica: 6caf99b397b6496da912b940275b8e12, State: Running, Role: LEADER
I20260806 15:58:14.307426 11268 consensus_queue.cc:237] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } }
I20260806 15:58:14.310585 10506 catalog_manager.cc:5495] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6caf99b397b6496da912b940275b8e12 (127.13.223.2). New cstate: current_term: 1 leader_uuid: "6caf99b397b6496da912b940275b8e12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6caf99b397b6496da912b940275b8e12" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 } health_report { overall_health: UNKNOWN } } }
I20260806 15:58:14.330395 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.326799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.13.223.2:45689)
0806 15:58:14.326921 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.326925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.326938 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:58:14.327001 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.327003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.327053 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.327138 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:14.327390 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.327966 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.328880 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.329103 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.329182 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.329474 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:14.329980 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:14.330094 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.330228 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.330265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:58:14.336265 11164 raft_consensus.cc:1261] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 1 FOLLOWER]: Refusing update from remote peer 6caf99b397b6496da912b940275b8e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:14.336264 10732 raft_consensus.cc:1261] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b [term 1 FOLLOWER]: Refusing update from remote peer 6caf99b397b6496da912b940275b8e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:14.337006 11272 consensus_queue.cc:1035] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97b48ba7a6e840a8bce668ca25bfb105" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:14.337209 11268 consensus_queue.cc:1035] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47ae3d43724942dea3a837bbcb61601b" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:14.346115 11278 mvcc.cc:205] Tried to move back new op lower bound from 7315586639194263552 to 7315586639085596672. Current Snapshot: MvccSnapshot[applied={T|T < 7315586639194263552}]
I20260806 15:58:14.353364 10507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:14.353560 10507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:14.356434 10507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a332b82e69340be853eb9619df26aea: Sending DeleteTablet for 3 replicas of tablet 132a66eb862d48f9b28a908c3e4e1c0f
I20260806 15:58:14.357398 10703 tablet_service.cc:1514] Processing DeleteTablet for tablet 132a66eb862d48f9b28a908c3e4e1c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:14 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:40816
I20260806 15:58:14.357604 10913 tablet_service.cc:1514] Processing DeleteTablet for tablet 132a66eb862d48f9b28a908c3e4e1c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:14 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:39706
I20260806 15:58:14.357628 11128 tablet_service.cc:1514] Processing DeleteTablet for tablet 132a66eb862d48f9b28a908c3e4e1c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:14 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:48492
I20260806 15:58:14.358202 11288 tablet_replica.cc:330] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: stopping tablet replica
I20260806 15:58:14.358398 11286 tablet_replica.cc:330] T 132a66eb862d48f9b28a908c3e4e1c0f P 47ae3d43724942dea3a837bbcb61601b: stopping tablet replica
I20260806 15:58:14.358592 11288 raft_consensus.cc:2229] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:14.358955 11288 raft_consensus.cc:2258] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:14.359014 11287 tablet_replica.cc:330] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: stopping tablet replica
I20260806 15:58:14.359071 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 10601
I20260806 15:58:14.359356 11287 raft_consensus.cc:2229] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:58:14.359769 11287 raft_consensus.cc:2258] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:14.363830 11288 ts_tablet_manager.cc:1905] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:14.363831 11287 ts_tablet_manager.cc:1905] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:14.366978 11288 ts_tablet_manager.cc:1918] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:14.366978 11287 ts_tablet_manager.cc:1918] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:14.367084 11288 log.cc:1196] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/132a66eb862d48f9b28a908c3e4e1c0f
I20260806 15:58:14.367084 11287 log.cc:1196] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/132a66eb862d48f9b28a908c3e4e1c0f
I20260806 15:58:14.367452 11287 ts_tablet_manager.cc:1939] T 132a66eb862d48f9b28a908c3e4e1c0f P 6caf99b397b6496da912b940275b8e12: Deleting consensus metadata
I20260806 15:58:14.367452 11288 ts_tablet_manager.cc:1939] T 132a66eb862d48f9b28a908c3e4e1c0f P 97b48ba7a6e840a8bce668ca25bfb105: Deleting consensus metadata
I20260806 15:58:14.368777 10479 catalog_manager.cc:4841] TS 6caf99b397b6496da912b940275b8e12 (127.13.223.2:45689): tablet 132a66eb862d48f9b28a908c3e4e1c0f (table test-table [id=091bf72f541745aca4578492ccbe955d]) successfully deleted
I20260806 15:58:14.368777 10478 catalog_manager.cc:4841] TS 97b48ba7a6e840a8bce668ca25bfb105 (127.13.223.3:37147): tablet 132a66eb862d48f9b28a908c3e4e1c0f (table test-table [id=091bf72f541745aca4578492ccbe955d]) successfully deleted
W20260806 15:58:14.391394 10477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:36125 (error 108)
I20260806 15:58:14.392307 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 10816
I20260806 15:58:14.397964 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.392374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:36125 (local address 127.0.0.1:40820)
0806 15:58:14.397657 (+ 5283us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.397790 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5127}
W20260806 15:58:14.399411 10477 catalog_manager.cc:4584] TS 47ae3d43724942dea3a837bbcb61601b (127.13.223.1:36125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 132a66eb862d48f9b28a908c3e4e1c0f: Network error: Client connection negotiation failed: client connection to 127.13.223.1:36125: connect: Connection refused (error 111)
I20260806 15:58:14.418447 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 11029
I20260806 15:58:14.432742 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 10453
2026-08-06T15:58:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2655 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T15:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:14Z Disabled control of system clock
I20260806 15:58:14.470039 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:38229
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:38229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 15:58:14.585681 11297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:14.585958 11297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:14.585994 11297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:14.589983 11297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:14.590051 11297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:14.590066 11297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:14.590085 11297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:14.590099 11297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:14.594959 11297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:38229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:38229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:14.596026 11297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:14.597378 11297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:14.606333 11307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:14.606333 11306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:14.606333 11309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:14.606559 11297 server_base.cc:1034] running on GCE node
I20260806 15:58:14.607270 11297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:14.608376 11297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:14.609546 11297 hybrid_clock.cc:648] HybridClock initialized: now 1786031894609523 us; error 28 us; skew 500 ppm
I20260806 15:58:14.611626 11297 webserver.cc:460] Webserver started at http://127.13.223.62:46525/ using document root <none> and password file <none>
I20260806 15:58:14.612250 11297 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:14.612337 11297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:14.612648 11297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:14.614814 11297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "f052b8a6eda64d47a48e3169653fc866"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "e11d36fe6e54d16ad785bb5ba351580d"
server_key_iv: "00a8c2cdc4b81fddaa6302f239a47fee"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.615406 11297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "f052b8a6eda64d47a48e3169653fc866"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "e11d36fe6e54d16ad785bb5ba351580d"
server_key_iv: "00a8c2cdc4b81fddaa6302f239a47fee"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.619356 11297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:58:14.621944 11319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.623212 11297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:14.623358 11297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "f052b8a6eda64d47a48e3169653fc866"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "e11d36fe6e54d16ad785bb5ba351580d"
server_key_iv: "00a8c2cdc4b81fddaa6302f239a47fee"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.623482 11297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:14.642391 11297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:14.643162 11297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:14.643373 11297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:14.652164 11297 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:38229
I20260806 15:58:14.652191 11403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:38229 every 8 connection(s)
I20260806 15:58:14.653142 11297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:14.656574 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:14.656869 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 11297
I20260806 15:58:14.656999 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:14.657305 14204 external_mini_cluster.cc:1428] Setting key cb371cd4447efb40fdaf9171897b7227
I20260806 15:58:14.664544 11404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866: Bootstrap starting.
I20260806 15:58:14.667109 11404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:14.667941 11404 log.cc:824] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:14.668220 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.658764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.13.223.62:38229)
0806 15:58:14.659547 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.659572 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.659622 (+ 50us) server_negotiation.cc:407] Connection header received
0806 15:58:14.660499 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.660556 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.661110 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.661517 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:14.662025 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.663704 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.664713 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.665200 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.665388 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.666394 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:14.666399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:14.666412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:14.666817 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:14.666876 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:14.666896 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:14.667004 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.667164 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.667455 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260806 15:58:14.670392 11404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866: No bootstrap required, opened a new log
I20260806 15:58:14.674304 11404 raft_consensus.cc:348] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f052b8a6eda64d47a48e3169653fc866" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38229 } }
I20260806 15:58:14.674594 11404 raft_consensus.cc:374] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:14.674681 11404 raft_consensus.cc:729] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f052b8a6eda64d47a48e3169653fc866, State: Initialized, Role: FOLLOWER
I20260806 15:58:14.675166 11404 consensus_queue.cc:260] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f052b8a6eda64d47a48e3169653fc866" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38229 } }
I20260806 15:58:14.675330 11404 raft_consensus.cc:388] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:14.675418 11404 raft_consensus.cc:482] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:14.675539 11404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:14.676764 11404 raft_consensus.cc:504] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f052b8a6eda64d47a48e3169653fc866" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38229 } }
I20260806 15:58:14.677129 11404 leader_election.cc:302] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f052b8a6eda64d47a48e3169653fc866; no voters:
I20260806 15:58:14.677492 11404 leader_election.cc:288] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:14.677629 11410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:14.677879 11410 raft_consensus.cc:686] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [term 1 LEADER]: Becoming Leader. State: Replica: f052b8a6eda64d47a48e3169653fc866, State: Running, Role: LEADER
I20260806 15:58:14.678174 11410 consensus_queue.cc:237] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f052b8a6eda64d47a48e3169653fc866" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38229 } }
I20260806 15:58:14.678799 11404 sys_catalog.cc:564] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:14.679925 11412 sys_catalog.cc:455] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f052b8a6eda64d47a48e3169653fc866. Latest consensus state: current_term: 1 leader_uuid: "f052b8a6eda64d47a48e3169653fc866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f052b8a6eda64d47a48e3169653fc866" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38229 } } }
I20260806 15:58:14.680104 11412 sys_catalog.cc:458] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:14.680371 11411 sys_catalog.cc:455] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f052b8a6eda64d47a48e3169653fc866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f052b8a6eda64d47a48e3169653fc866" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38229 } } }
I20260806 15:58:14.680486 11411 sys_catalog.cc:458] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:14.680649 11421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:14.683822 11421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:14.689272 11421 catalog_manager.cc:1347] Generated new cluster ID: 4eba7ea8c3d145b1b9a477e4aa04cc3c
I20260806 15:58:14.689347 11421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:14.704042 11421 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:14.705313 11421 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:14.714579 11421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866: Generated new TSK 0
I20260806 15:58:14.715466 11421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:14.719868 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38229
--builtin_ntp_servers=127.13.223.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 15:58:14.836297 11435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:14.836711 11435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:14.836802 11435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:14.840950 11435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:14.841032 11435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:14.841130 11435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:58:14.846011 11435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:14.847175 11435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:14.848620 11435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:14.858332 11444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:14.858400 11443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:14.858551 11435 server_base.cc:1034] running on GCE node
W20260806 15:58:14.858695 11446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:14.859081 11435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:14.859730 11435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:14.860910 11435 hybrid_clock.cc:648] HybridClock initialized: now 1786031894860884 us; error 41 us; skew 500 ppm
I20260806 15:58:14.862699 11435 webserver.cc:460] Webserver started at http://127.13.223.1:35921/ using document root <none> and password file <none>
I20260806 15:58:14.863279 11435 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:14.863366 11435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:14.863590 11435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:14.865692 11435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "e65a7cb0d5df41e999dab7612ad1f08f"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "8cd2db30a9c85d6d3792df8b33943819"
server_key_iv: "c8a8a78f43e05242318bea860456c922"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.866267 11435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "e65a7cb0d5df41e999dab7612ad1f08f"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "8cd2db30a9c85d6d3792df8b33943819"
server_key_iv: "c8a8a78f43e05242318bea860456c922"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.870004 11435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:58:14.872422 11454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.873651 11435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:14.873797 11435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "e65a7cb0d5df41e999dab7612ad1f08f"
format_stamp: "Formatted at 2026-08-06 15:58:14 on dist-test-slave-rwrg"
server_key: "8cd2db30a9c85d6d3792df8b33943819"
server_key_iv: "c8a8a78f43e05242318bea860456c922"
server_key_version: "encryptionkey@0"
I20260806 15:58:14.873909 11435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:14.891418 11435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:14.892113 11435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:14.892316 11435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:14.892946 11435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:14.893887 11435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:14.893955 11435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.894024 11435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:14.894063 11435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:14.903674 11435 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:43877
I20260806 15:58:14.903699 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:43877 every 8 connection(s)
I20260806 15:58:14.904642 11435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:58:14.908293 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 11435
I20260806 15:58:14.908425 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:58:14.908798 14204 external_mini_cluster.cc:1428] Setting key a6f8f11a83e277471db8f5a119be1233
I20260806 15:58:14.910921 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38229
--builtin_ntp_servers=127.13.223.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 15:58:14.916050 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.906487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:47743 (local address 127.13.223.62:38229)
0806 15:58:14.906679 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:14.906683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:14.907809 (+ 1126us) server_negotiation.cc:407] Connection header received
0806 15:58:14.909753 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:14.909756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:14.909809 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:14.909890 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:14.911078 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.911702 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.913328 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.913767 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.913898 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.915274 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:14.915280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:14.915286 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:14.915318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:14.915346 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:14.915359 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:14.915446 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:14.915735 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 15:58:14.915820 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 15:58:14.916936 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:14.906972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38229 (local address 127.13.223.1:47743)
0806 15:58:14.907655 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:14.907700 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:14.909525 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:14.910083 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:14.910092 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:14.910200 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:14.910903 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.910922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.911839 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.911846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.913176 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:14.913187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:14.913924 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:14.913927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:14.914082 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:14.914961 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:14.914963 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:14.915007 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:14.915083 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:14.915501 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:14.915515 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:14.915522 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 15:58:14.915695 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 15:58:14.915955 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260806 15:58:14.918294 11641 heartbeater.cc:347] Connected to a master server at 127.13.223.62:38229
I20260806 15:58:14.918567 11641 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:14.919327 11641 heartbeater.cc:511] Master 127.13.223.62:38229 requested a full tablet report, sending...
I20260806 15:58:14.921088 11348 ts_manager.cc:194] Registered new tserver with Master: e65a7cb0d5df41e999dab7612ad1f08f (127.13.223.1:43877)
I20260806 15:58:14.922369 11348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.1:47743
W20260806 15:58:15.030227 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:15.030512 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:15.030555 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:15.034623 11644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:15.034693 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:15.034749 11644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:58:15.040306 11644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:15.041631 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:15.043114 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:15.052933 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:15.052912 11651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:15.052917 11653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:15.053125 11644 server_base.cc:1034] running on GCE node
I20260806 15:58:15.053637 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:15.054358 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:15.055538 11644 hybrid_clock.cc:648] HybridClock initialized: now 1786031895055510 us; error 43 us; skew 500 ppm
I20260806 15:58:15.057494 11644 webserver.cc:460] Webserver started at http://127.13.223.2:44629/ using document root <none> and password file <none>
I20260806 15:58:15.058084 11644 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:15.058147 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:15.058388 11644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:15.060412 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "1d05d0ffaf6b439393c9298d3d94887e"
format_stamp: "Formatted at 2026-08-06 15:58:15 on dist-test-slave-rwrg"
server_key: "b260fc9817f261120cfee195023ec089"
server_key_iv: "0de9629de0764a7d281937f65aa42709"
server_key_version: "encryptionkey@0"
I20260806 15:58:15.061067 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "1d05d0ffaf6b439393c9298d3d94887e"
format_stamp: "Formatted at 2026-08-06 15:58:15 on dist-test-slave-rwrg"
server_key: "b260fc9817f261120cfee195023ec089"
server_key_iv: "0de9629de0764a7d281937f65aa42709"
server_key_version: "encryptionkey@0"
I20260806 15:58:15.064821 11644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:58:15.067340 11662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:15.068745 11644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:15.068892 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "1d05d0ffaf6b439393c9298d3d94887e"
format_stamp: "Formatted at 2026-08-06 15:58:15 on dist-test-slave-rwrg"
server_key: "b260fc9817f261120cfee195023ec089"
server_key_iv: "0de9629de0764a7d281937f65aa42709"
server_key_version: "encryptionkey@0"
I20260806 15:58:15.069007 11644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:15.089852 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:15.090653 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:15.090879 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:15.091547 11644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:15.092680 11644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:15.092736 11644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:15.092805 11644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:15.092850 11644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:15.103340 11644 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:34129
I20260806 15:58:15.103375 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:34129 every 8 connection(s)
I20260806 15:58:15.104300 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:58:15.108963 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 11644
I20260806 15:58:15.109117 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:58:15.109498 14204 external_mini_cluster.cc:1428] Setting key 984ad6b23dd84b3826d4cbbf2814eaa3
I20260806 15:58:15.111828 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38229
--builtin_ntp_servers=127.13.223.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 15:58:15.115293 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.106164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:51219 (local address 127.13.223.62:38229)
0806 15:58:15.106328 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.106332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.107511 (+ 1179us) server_negotiation.cc:407] Connection header received
0806 15:58:15.109108 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.109112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.109184 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.109286 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:15.110479 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.111148 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.112909 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.113135 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.113229 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.114678 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:15.114685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:15.114691 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:15.114711 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:15.114739 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:15.114752 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:15.114849 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.115000 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.115084 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:58:15.116072 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.106630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38229 (local address 127.13.223.2:51219)
0806 15:58:15.107333 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.107385 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.108804 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.109502 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.109513 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.109630 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:15.110292 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.110314 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.111269 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.111272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.112750 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.112765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.113304 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.113310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.113553 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.114343 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:15.114345 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:15.114388 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:15.114457 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:15.114848 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:15.114855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:15.114859 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.114957 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.115205 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":127,"threads_started":1}
I20260806 15:58:15.117379 11833 heartbeater.cc:347] Connected to a master server at 127.13.223.62:38229
I20260806 15:58:15.117621 11833 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:15.118166 11833 heartbeater.cc:511] Master 127.13.223.62:38229 requested a full tablet report, sending...
I20260806 15:58:15.119264 11348 ts_manager.cc:194] Registered new tserver with Master: 1d05d0ffaf6b439393c9298d3d94887e (127.13.223.2:34129)
I20260806 15:58:15.119829 11348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.2:51219
W20260806 15:58:15.231623 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:15.231889 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:15.231927 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:15.236068 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:15.236130 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:15.236184 11838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:58:15.241008 11838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:15.242053 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:15.243432 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:15.253140 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:15.253175 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:15.253201 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:15.253336 11838 server_base.cc:1034] running on GCE node
I20260806 15:58:15.253813 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:15.254500 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:15.255681 11838 hybrid_clock.cc:648] HybridClock initialized: now 1786031895255649 us; error 46 us; skew 500 ppm
I20260806 15:58:15.257736 11838 webserver.cc:460] Webserver started at http://127.13.223.3:44783/ using document root <none> and password file <none>
I20260806 15:58:15.258323 11838 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:15.258414 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:15.258628 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:15.260782 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "ec6adce906ae4273bf121adf63f80848"
format_stamp: "Formatted at 2026-08-06 15:58:15 on dist-test-slave-rwrg"
server_key: "c3e677d6550f0c77606f0b9d0117272c"
server_key_iv: "cccd3258aa108a6a195797c768361c51"
server_key_version: "encryptionkey@0"
I20260806 15:58:15.261365 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "ec6adce906ae4273bf121adf63f80848"
format_stamp: "Formatted at 2026-08-06 15:58:15 on dist-test-slave-rwrg"
server_key: "c3e677d6550f0c77606f0b9d0117272c"
server_key_iv: "cccd3258aa108a6a195797c768361c51"
server_key_version: "encryptionkey@0"
I20260806 15:58:15.265596 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:58:15.268172 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:15.269603 11838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:15.269754 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "ec6adce906ae4273bf121adf63f80848"
format_stamp: "Formatted at 2026-08-06 15:58:15 on dist-test-slave-rwrg"
server_key: "c3e677d6550f0c77606f0b9d0117272c"
server_key_iv: "cccd3258aa108a6a195797c768361c51"
server_key_version: "encryptionkey@0"
I20260806 15:58:15.269865 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:15.282599 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:15.283231 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:15.283421 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:15.283984 11838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:15.284965 11838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:15.285032 11838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:15.285097 11838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:15.285136 11838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:15.294435 11838 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:35497
I20260806 15:58:15.294487 12048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:35497 every 8 connection(s)
I20260806 15:58:15.295346 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:58:15.300136 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 11838
I20260806 15:58:15.300262 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:58:15.300597 14204 external_mini_cluster.cc:1428] Setting key e9cc5dfc7f25265d4a4521b72b3d0d06
I20260806 15:58:15.304584 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.297142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:54545 (local address 127.13.223.62:38229)
0806 15:58:15.297281 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.297285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.298297 (+ 1012us) server_negotiation.cc:407] Connection header received
0806 15:58:15.299658 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.299661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.299720 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.299817 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:15.300920 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.301521 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.302648 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.302866 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.302974 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.304069 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:15.304072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:15.304075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:15.304087 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:15.304113 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:15.304120 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:15.304195 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.304354 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.304414 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 15:58:15.305401 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.297529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38229 (local address 127.13.223.3:54545)
0806 15:58:15.298157 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.298196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.299457 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.300013 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.300020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.300109 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:15.300775 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.300793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.301649 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.301652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.302523 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.302533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.303015 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.303019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.303198 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.303816 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:15.303817 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:15.303856 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:15.303917 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:15.304197 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:15.304209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:15.304218 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.304349 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.304665 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260806 15:58:15.306497 12049 heartbeater.cc:347] Connected to a master server at 127.13.223.62:38229
I20260806 15:58:15.306730 12049 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:15.307224 12049 heartbeater.cc:511] Master 127.13.223.62:38229 requested a full tablet report, sending...
I20260806 15:58:15.308272 11348 ts_manager.cc:194] Registered new tserver with Master: ec6adce906ae4273bf121adf63f80848 (127.13.223.3:35497)
I20260806 15:58:15.308892 11348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.3:54545
I20260806 15:58:15.314255 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:58:15.320590 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.316331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.13.223.62:38229)
0806 15:58:15.316526 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.316531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.316683 (+ 152us) server_negotiation.cc:407] Connection header received
0806 15:58:15.316834 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.316837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.316884 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.316978 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:15.317277 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.317849 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.318725 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.318949 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.319058 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.319979 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:15.319987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:15.319990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:15.320002 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:15.320021 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:15.320029 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:15.320185 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.320329 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.320381 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 15:58:15.323372 11348 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:58:15.326491 11348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:58:15.341262 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.335240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.13.223.2:34129)
0806 15:58:15.335733 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.335749 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.335776 (+ 27us) server_negotiation.cc:407] Connection header received
0806 15:58:15.336479 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.336485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.336770 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.336909 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:15.337365 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.338172 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.339830 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.340433 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.340613 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.340816 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.340953 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.341079 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":133,"threads_started":1}
I20260806 15:58:15.341395 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.335788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:43877 (local address 127.0.0.1:41802)
0806 15:58:15.336712 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.336727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.336824 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.337296 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.337299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.337320 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:15.337424 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.337429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.338754 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.338760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.339863 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.339873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.340895 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.340901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.340999 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.341021 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.341092 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.341245 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":4,"thread_start_us":92,"threads_started":1}
I20260806 15:58:15.341428 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.336100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.13.223.1:43877)
0806 15:58:15.336573 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.336585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.336920 (+ 335us) server_negotiation.cc:407] Connection header received
0806 15:58:15.337004 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.337008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.337144 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.337312 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:15.337551 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.338642 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.339989 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.340776 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.340864 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.341005 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.341119 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.341225 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":121,"threads_started":1}
I20260806 15:58:15.341895 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.335099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:34129 (local address 127.0.0.1:54342)
0806 15:58:15.335445 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.335498 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.336305 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.337090 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.337093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.337110 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:15.337222 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.337228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.338294 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.338296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.339697 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.339708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.340616 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.340620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.340728 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.340766 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.341420 (+ 654us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.341711 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":32,"thread_start_us":73,"threads_started":1}
I20260806 15:58:15.341910 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.335470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:35497 (local address 127.0.0.1:58998)
0806 15:58:15.335922 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.335936 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.336075 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.336914 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.336920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.336942 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:15.337097 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.337104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.338543 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.338547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.339697 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.339708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.341087 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.341090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.341236 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.341260 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.341560 (+ 300us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.341739 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":133,"thread_start_us":108,"threads_started":1}
I20260806 15:58:15.342321 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.335974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.13.223.3:35497)
0806 15:58:15.336319 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.336330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.336356 (+ 26us) server_negotiation.cc:407] Connection header received
0806 15:58:15.336437 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.336443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.336708 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.336867 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:15.337220 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.338426 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.339870 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.340693 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.340809 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.341005 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.341675 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.341792 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
I20260806 15:58:15.344718 11541 tablet_service.cc:1467] Processing CreateTablet for tablet ac6bbbfab3074d1ba8f6f8f10b317381 (DEFAULT_TABLE table=test-table [id=d55723cea1e74650987c5b0a760bfc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:15.345067 11735 tablet_service.cc:1467] Processing CreateTablet for tablet ac6bbbfab3074d1ba8f6f8f10b317381 (DEFAULT_TABLE table=test-table [id=d55723cea1e74650987c5b0a760bfc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:15.345081 11941 tablet_service.cc:1467] Processing CreateTablet for tablet ac6bbbfab3074d1ba8f6f8f10b317381 (DEFAULT_TABLE table=test-table [id=d55723cea1e74650987c5b0a760bfc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:15.345801 11541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6bbbfab3074d1ba8f6f8f10b317381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:15.346127 11941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6bbbfab3074d1ba8f6f8f10b317381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:15.346127 11735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6bbbfab3074d1ba8f6f8f10b317381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:15.352762 12075 tablet_bootstrap.cc:492] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: Bootstrap starting.
I20260806 15:58:15.353925 12078 tablet_bootstrap.cc:492] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Bootstrap starting.
I20260806 15:58:15.355682 12075 tablet_bootstrap.cc:654] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:15.356117 12078 tablet_bootstrap.cc:654] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:15.356684 12075 log.cc:824] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:15.356863 12079 tablet_bootstrap.cc:492] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Bootstrap starting.
I20260806 15:58:15.357106 12078 log.cc:824] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:15.358966 12078 tablet_bootstrap.cc:492] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: No bootstrap required, opened a new log
I20260806 15:58:15.359061 12075 tablet_bootstrap.cc:492] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: No bootstrap required, opened a new log
I20260806 15:58:15.359031 12079 tablet_bootstrap.cc:654] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:15.359179 12078 ts_tablet_manager.cc:1397] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 15:58:15.359359 12075 ts_tablet_manager.cc:1397] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260806 15:58:15.359956 12079 log.cc:824] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:15.362114 12079 tablet_bootstrap.cc:492] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: No bootstrap required, opened a new log
I20260806 15:58:15.362308 12079 ts_tablet_manager.cc:1397] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:58:15.362829 12078 raft_consensus.cc:348] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.363080 12078 raft_consensus.cc:374] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:15.363134 12078 raft_consensus.cc:729] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d05d0ffaf6b439393c9298d3d94887e, State: Initialized, Role: FOLLOWER
I20260806 15:58:15.363572 12078 consensus_queue.cc:260] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.363710 12075 raft_consensus.cc:348] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.363960 12075 raft_consensus.cc:374] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:15.364013 12075 raft_consensus.cc:729] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65a7cb0d5df41e999dab7612ad1f08f, State: Initialized, Role: FOLLOWER
I20260806 15:58:15.364388 11833 heartbeater.cc:503] Master 127.13.223.62:38229 was elected leader, sending a full tablet report...
I20260806 15:58:15.364475 12075 consensus_queue.cc:260] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.364625 12078 ts_tablet_manager.cc:1428] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 15:58:15.365595 12075 ts_tablet_manager.cc:1428] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 15:58:15.366515 12079 raft_consensus.cc:348] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.366700 12079 raft_consensus.cc:374] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:15.366753 12079 raft_consensus.cc:729] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6adce906ae4273bf121adf63f80848, State: Initialized, Role: FOLLOWER
I20260806 15:58:15.367154 12079 consensus_queue.cc:260] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.367592 11641 heartbeater.cc:503] Master 127.13.223.62:38229 was elected leader, sending a full tablet report...
I20260806 15:58:15.367758 12049 heartbeater.cc:503] Master 127.13.223.62:38229 was elected leader, sending a full tablet report...
I20260806 15:58:15.368984 12079 ts_tablet_manager.cc:1428] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 15:58:15.405813 11642 tablet.cc:2369] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 15:58:15.546325 12050 tablet.cc:2369] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:58:15.548137 12088 raft_consensus.cc:482] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:15.548355 12088 raft_consensus.cc:504] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.549573 12088 leader_election.cc:288] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d05d0ffaf6b439393c9298d3d94887e (127.13.223.2:34129), ec6adce906ae4273bf121adf63f80848 (127.13.223.3:35497)
I20260806 15:58:15.553587 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.549998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:53759 (local address 127.13.223.3:35497)
0806 15:58:15.550132 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.550136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.550432 (+ 296us) server_negotiation.cc:407] Connection header received
0806 15:58:15.550572 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.550576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.550659 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.550741 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:15.551029 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.551522 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.552632 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.553025 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.553161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.553213 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.553386 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.553459 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:58:15.553615 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.549897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:34129 (local address 127.13.223.1:54291)
0806 15:58:15.550213 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.550227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.550329 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.550599 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.550602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.550620 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:15.550784 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.550791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.551556 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.551563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.552441 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.552451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.553181 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.553184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.553286 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.553303 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.553388 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.553459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260806 15:58:15.553608 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.549998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:54291 (local address 127.13.223.2:34129)
0806 15:58:15.550242 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.550247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.550291 (+ 44us) server_negotiation.cc:407] Connection header received
0806 15:58:15.550442 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.550446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.550502 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.550582 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:15.550910 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.551448 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.552620 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.553017 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.553163 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.553213 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.553421 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.553492 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260806 15:58:15.553675 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.549897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:35497 (local address 127.13.223.1:53759)
0806 15:58:15.550303 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.550318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:15.550447 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:15.550766 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:15.550768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:15.550783 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:15.550919 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.550926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.551633 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.551635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.552437 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:15.552446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.553153 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.553157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:15.553254 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.553281 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 15:58:15.553347 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 15:58:15.553433 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":54,"thread_start_us":88,"threads_started":1}
I20260806 15:58:15.554294 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6bbbfab3074d1ba8f6f8f10b317381" candidate_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6adce906ae4273bf121adf63f80848" is_pre_election: true
I20260806 15:58:15.554302 11766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6bbbfab3074d1ba8f6f8f10b317381" candidate_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d05d0ffaf6b439393c9298d3d94887e" is_pre_election: true
I20260806 15:58:15.554594 11979 raft_consensus.cc:2454] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65a7cb0d5df41e999dab7612ad1f08f in term 0.
I20260806 15:58:15.554610 11766 raft_consensus.cc:2454] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65a7cb0d5df41e999dab7612ad1f08f in term 0.
I20260806 15:58:15.555184 11462 leader_election.cc:302] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e65a7cb0d5df41e999dab7612ad1f08f, ec6adce906ae4273bf121adf63f80848; no voters:
I20260806 15:58:15.555492 12088 raft_consensus.cc:2781] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:58:15.555575 12088 raft_consensus.cc:482] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:58:15.555629 12088 raft_consensus.cc:3037] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:15.556829 12088 raft_consensus.cc:504] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.557236 12088 leader_election.cc:288] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [CANDIDATE]: Term 1 election: Requested vote from peers 1d05d0ffaf6b439393c9298d3d94887e (127.13.223.2:34129), ec6adce906ae4273bf121adf63f80848 (127.13.223.3:35497)
I20260806 15:58:15.557570 11766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6bbbfab3074d1ba8f6f8f10b317381" candidate_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d05d0ffaf6b439393c9298d3d94887e"
I20260806 15:58:15.557664 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6bbbfab3074d1ba8f6f8f10b317381" candidate_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6adce906ae4273bf121adf63f80848"
I20260806 15:58:15.557706 11766 raft_consensus.cc:3037] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:15.557775 11979 raft_consensus.cc:3037] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:15.559029 11979 raft_consensus.cc:2454] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65a7cb0d5df41e999dab7612ad1f08f in term 1.
I20260806 15:58:15.559029 11766 raft_consensus.cc:2454] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65a7cb0d5df41e999dab7612ad1f08f in term 1.
I20260806 15:58:15.559360 11456 leader_election.cc:302] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d05d0ffaf6b439393c9298d3d94887e, e65a7cb0d5df41e999dab7612ad1f08f; no voters:
I20260806 15:58:15.559514 12088 raft_consensus.cc:2781] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:15.559770 12088 raft_consensus.cc:686] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [term 1 LEADER]: Becoming Leader. State: Replica: e65a7cb0d5df41e999dab7612ad1f08f, State: Running, Role: LEADER
I20260806 15:58:15.560115 12088 consensus_queue.cc:237] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } }
I20260806 15:58:15.563298 11347 catalog_manager.cc:5495] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f reported cstate change: term changed from 0 to 1, leader changed from <none> to e65a7cb0d5df41e999dab7612ad1f08f (127.13.223.1). New cstate: current_term: 1 leader_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e65a7cb0d5df41e999dab7612ad1f08f" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 43877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 } health_report { overall_health: UNKNOWN } } }
I20260806 15:58:15.578730 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.575078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.13.223.1:43877)
0806 15:58:15.575214 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:15.575218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:15.575233 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:58:15.575360 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:15.575363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:15.575425 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:15.575528 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:15.575771 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.576257 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.577263 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:15.577493 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:15.577594 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:15.577922 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:15.578318 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:15.578466 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:15.578560 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 15:58:15.578594 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:58:15.584685 11979 raft_consensus.cc:1261] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 1 FOLLOWER]: Refusing update from remote peer e65a7cb0d5df41e999dab7612ad1f08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:15.584699 11766 raft_consensus.cc:1261] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 1 FOLLOWER]: Refusing update from remote peer e65a7cb0d5df41e999dab7612ad1f08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:15.585352 12097 consensus_queue.cc:1035] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6adce906ae4273bf121adf63f80848" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:15.585544 12088 consensus_queue.cc:1035] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d05d0ffaf6b439393c9298d3d94887e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:15.594142 12104 mvcc.cc:205] Tried to move back new op lower bound from 7315586644307914752 to 7315586644217004032. Current Snapshot: MvccSnapshot[applied={T|T < 7315586644307914752}]
I20260806 15:58:15.601176 11344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:15.601342 11344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:15.603569 11344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f052b8a6eda64d47a48e3169653fc866: Sending DeleteTablet for 3 replicas of tablet ac6bbbfab3074d1ba8f6f8f10b317381
I20260806 15:58:15.604457 11735 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6bbbfab3074d1ba8f6f8f10b317381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:15 UTC) from {username='slave'} at 127.0.0.1:54342
I20260806 15:58:15.604645 11541 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6bbbfab3074d1ba8f6f8f10b317381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:15 UTC) from {username='slave'} at 127.0.0.1:41802
I20260806 15:58:15.604998 11941 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6bbbfab3074d1ba8f6f8f10b317381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:15 UTC) from {username='slave'} at 127.0.0.1:58998
I20260806 15:58:15.605400 12116 tablet_replica.cc:330] T ac6bbbfab3074d1ba8f6f8f10b317381 P e65a7cb0d5df41e999dab7612ad1f08f: stopping tablet replica
I20260806 15:58:15.605504 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 11435
I20260806 15:58:15.610881 12115 tablet_replica.cc:330] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: stopping tablet replica
I20260806 15:58:15.611217 12115 raft_consensus.cc:2229] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:15.611518 12115 raft_consensus.cc:2258] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:15.611841 12117 tablet_replica.cc:330] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: stopping tablet replica
I20260806 15:58:15.612098 12117 raft_consensus.cc:2229] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:15.612337 12117 raft_consensus.cc:2258] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:15.614205 12117 ts_tablet_manager.cc:1905] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:15.614348 12115 ts_tablet_manager.cc:1905] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:15.616626 12117 ts_tablet_manager.cc:1918] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:15.616747 12117 log.cc:1196] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/ac6bbbfab3074d1ba8f6f8f10b317381
I20260806 15:58:15.617028 12117 ts_tablet_manager.cc:1939] T ac6bbbfab3074d1ba8f6f8f10b317381 P ec6adce906ae4273bf121adf63f80848: Deleting consensus metadata
I20260806 15:58:15.617172 12115 ts_tablet_manager.cc:1918] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:15.617272 12115 log.cc:1196] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/ac6bbbfab3074d1ba8f6f8f10b317381
I20260806 15:58:15.617566 12115 ts_tablet_manager.cc:1939] T ac6bbbfab3074d1ba8f6f8f10b317381 P 1d05d0ffaf6b439393c9298d3d94887e: Deleting consensus metadata
I20260806 15:58:15.618317 11324 catalog_manager.cc:4841] TS ec6adce906ae4273bf121adf63f80848 (127.13.223.3:35497): tablet ac6bbbfab3074d1ba8f6f8f10b317381 (table test-table [id=d55723cea1e74650987c5b0a760bfc9b]) successfully deleted
I20260806 15:58:15.618700 11323 catalog_manager.cc:4841] TS 1d05d0ffaf6b439393c9298d3d94887e (127.13.223.2:34129): tablet ac6bbbfab3074d1ba8f6f8f10b317381 (table test-table [id=d55723cea1e74650987c5b0a760bfc9b]) successfully deleted
W20260806 15:58:15.620198 11323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:43877 (error 108)
I20260806 15:58:15.620311 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 11644
I20260806 15:58:15.625116 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:15.624744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:43877 (local address 127.0.0.1:41820)
0806 15:58:15.624911 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:15.624998 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 15:58:15.625315 11323 catalog_manager.cc:4584] TS e65a7cb0d5df41e999dab7612ad1f08f (127.13.223.1:43877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac6bbbfab3074d1ba8f6f8f10b317381: Network error: Client connection negotiation failed: client connection to 127.13.223.1:43877: connect: Connection refused (error 111)
I20260806 15:58:15.632643 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 11838
I20260806 15:58:15.645973 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 11297
2026-08-06T15:58:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:15 dist-test-slave-rwrg krb5kdc[12124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:15 dist-test-slave-rwrg krb5kdc[12124](info): set up 2 sockets
Aug 06 15:58:15 dist-test-slave-rwrg krb5kdc[12124](info): commencing operation
krb5kdc: starting...
W20260806 15:58:16.907315 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:16 dist-test-slave-rwrg krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:17.037731 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44335
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:44335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:17.153861 12141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:17.154143 12141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:17.154184 12141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:17.158216 12141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:17.158280 12141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:17.158300 12141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:17.158317 12141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:17.158332 12141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:17.163849 12141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:44335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:44335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:17.164956 12141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:17.165947 12141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:17.174786 12150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.174794 12149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.174786 12153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:17.174826 12141 server_base.cc:1034] running on GCE node
I20260806 15:58:17.175652 12141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:17.176783 12141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:17.177963 12141 hybrid_clock.cc:648] HybridClock initialized: now 1786031897177934 us; error 42 us; skew 500 ppm
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:17.180933 12141 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:17.181864 12141 webserver.cc:460] Webserver started at http://127.13.223.62:40827/ using document root <none> and password file <none>
I20260806 15:58:17.182422 12141 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:17.182521 12141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:17.182750 12141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:17.184881 12141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "e615c8b04dad432882b1045c2b7825db"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "d81c83318a4c9e312d54d5b6a28a6d79"
server_key_iv: "b10fad3231e2f095663a33595865d8a4"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.185456 12141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "e615c8b04dad432882b1045c2b7825db"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "d81c83318a4c9e312d54d5b6a28a6d79"
server_key_iv: "b10fad3231e2f095663a33595865d8a4"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.189332 12141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 15:58:17.191705 12163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.193053 12141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:58:17.193199 12141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "e615c8b04dad432882b1045c2b7825db"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "d81c83318a4c9e312d54d5b6a28a6d79"
server_key_iv: "b10fad3231e2f095663a33595865d8a4"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.193315 12141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:17.204298 12141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:17.207449 12141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:17.207651 12141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:17.214917 12141 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:44335
I20260806 15:58:17.214946 12247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:44335 every 8 connection(s)
I20260806 15:58:17.215839 12141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:17.218493 12249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:17.223927 12249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: Bootstrap starting.
I20260806 15:58:17.224682 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 12141
I20260806 15:58:17.224818 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:17.225165 14204 external_mini_cluster.cc:1428] Setting key f236a91ba066b41b077eff9c88a04753
I20260806 15:58:17.226864 12249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:17.227607 12249 log.cc:824] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:17.229844 12249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: No bootstrap required, opened a new log
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:17.234469 12249 raft_consensus.cc:348] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e615c8b04dad432882b1045c2b7825db" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44335 } }
I20260806 15:58:17.234705 12249 raft_consensus.cc:374] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:17.234772 12249 raft_consensus.cc:729] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e615c8b04dad432882b1045c2b7825db, State: Initialized, Role: FOLLOWER
I20260806 15:58:17.235204 12249 consensus_queue.cc:260] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e615c8b04dad432882b1045c2b7825db" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44335 } }
I20260806 15:58:17.235364 12249 raft_consensus.cc:388] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:17.235447 12249 raft_consensus.cc:482] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:17.235561 12249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:17.236430 12249 raft_consensus.cc:504] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e615c8b04dad432882b1045c2b7825db" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44335 } }
I20260806 15:58:17.236801 12249 leader_election.cc:302] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e615c8b04dad432882b1045c2b7825db; no voters:
I20260806 15:58:17.237133 12249 leader_election.cc:288] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:17.237248 12254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:17.237437 12254 raft_consensus.cc:686] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [term 1 LEADER]: Becoming Leader. State: Replica: e615c8b04dad432882b1045c2b7825db, State: Running, Role: LEADER
I20260806 15:58:17.237732 12254 consensus_queue.cc:237] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e615c8b04dad432882b1045c2b7825db" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44335 } }
I20260806 15:58:17.238427 12249 sys_catalog.cc:564] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:17.238903 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [sys.catalog]: SysCatalogTable state changed. Reason: New leader e615c8b04dad432882b1045c2b7825db. Latest consensus state: current_term: 1 leader_uuid: "e615c8b04dad432882b1045c2b7825db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e615c8b04dad432882b1045c2b7825db" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44335 } } }
I20260806 15:58:17.239024 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:17.240365 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.226365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.13.223.62:44335)
0806 15:58:17.226774 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:17.226789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:17.226824 (+ 35us) server_negotiation.cc:407] Connection header received
0806 15:58:17.227534 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:17.227556 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:17.227942 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:17.228331 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:17.229218 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.230233 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.231390 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.231880 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.231996 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.234972 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:17.234977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:17.234990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:17.235052 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:17.237053 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.237609 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.237615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.237621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.237758 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.238082 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.238086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.238088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.238561 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:17.238775 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:17.239066 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 15:58:17.239387 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":129,"threads_started":1}
I20260806 15:58:17.241498 12255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e615c8b04dad432882b1045c2b7825db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e615c8b04dad432882b1045c2b7825db" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 44335 } } }
I20260806 15:58:17.241616 12255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db [sys.catalog]: This master's current role is: LEADER
W20260806 15:58:17.243610 12282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:17.243716 12282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:17.243793 12262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:17.244468 12262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:17.250005 12262 catalog_manager.cc:1347] Generated new cluster ID: d65bcb589b7f4cb5acb33c69402c2fcf
I20260806 15:58:17.250075 12262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:17.273751 12262 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:17.274755 12262 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:17.282021 12262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: Generated new TSK 0
I20260806 15:58:17.282752 12262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
Entry for principal kudu/127.13.223.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab.
I20260806 15:58:17.315644 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44335
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:17.437384 12289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:17.437675 12289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:17.437721 12289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:17.441951 12289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:17.442046 12289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:17.442112 12289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:58:17.447448 12289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.1.keytab
--principal=kudu/127.13.223.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:17.448563 12289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:17.449621 12289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:17.459729 12298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.459800 12296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.459800 12295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:17.459990 12289 server_base.cc:1034] running on GCE node
I20260806 15:58:17.460402 12289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:17.461093 12289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:17.462262 12289 hybrid_clock.cc:648] HybridClock initialized: now 1786031897462230 us; error 45 us; skew 500 ppm
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:17.465256 12289 init.cc:377] Logged in from keytab as kudu/127.13.223.1@KRBTEST.COM (short username kudu)
I20260806 15:58:17.466123 12289 webserver.cc:460] Webserver started at http://127.13.223.1:38691/ using document root <none> and password file <none>
I20260806 15:58:17.466670 12289 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:17.466747 12289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:17.466917 12289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:17.469048 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "085e37b576cab600fabde3443ad69577"
server_key_iv: "2a6178baf9d0147a0da498db5b6abeb8"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.469591 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "085e37b576cab600fabde3443ad69577"
server_key_iv: "2a6178baf9d0147a0da498db5b6abeb8"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.473402 12289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 15:58:17.475955 12306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.477301 12289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:17.477437 12289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "085e37b576cab600fabde3443ad69577"
server_key_iv: "2a6178baf9d0147a0da498db5b6abeb8"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.477548 12289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:17.486954 12289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:17.490332 12289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:17.490546 12289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:17.491184 12289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:17.492075 12289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:17.492125 12289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.492187 12289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:17.492219 12289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.501839 12289 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:37697
I20260806 15:58:17.501896 12482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:37697 every 8 connection(s)
I20260806 15:58:17.503155 12289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.1@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:17.513425 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 12289
I20260806 15:58:17.513566 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:58:17.513947 14204 external_mini_cluster.cc:1428] Setting key 22741d9f5ce09c2ad097c96e10fcbf5d
I20260806 15:58:17.517719 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.505219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:57551 (local address 127.13.223.62:44335)
0806 15:58:17.505367 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:17.505371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:17.506494 (+ 1123us) server_negotiation.cc:407] Connection header received
0806 15:58:17.507373 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:17.507376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:17.507424 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:17.507500 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:17.508738 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.509409 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.510569 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.510860 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.510945 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.513809 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:17.513813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:17.513817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:17.513847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:17.515341 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.515915 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.515921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.515925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.516018 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.516350 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.516356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.516361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.516673 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:17.516774 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:17.517334 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 15:58:17.517483 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 15:58:17.518662 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.505632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44335 (local address 127.13.223.1:57551)
0806 15:58:17.506319 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:17.506380 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:17.507177 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:17.507633 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:17.507639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:17.507994 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:17.508576 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:17.508597 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.509526 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.509528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:17.510431 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:17.510441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.511034 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.511038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:17.511196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.511884 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:17.511901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:17.513628 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:17.515476 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:17.515482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:17.515497 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:17.515800 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:17.516116 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:17.516118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:17.516120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:17.516258 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:17.516817 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:17.516822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:17.517068 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 15:58:17.517241 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 15:58:17.517669 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":108,"threads_started":1}
I20260806 15:58:17.519948 12483 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44335
I20260806 15:58:17.520212 12483 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:17.520785 12483 heartbeater.cc:511] Master 127.13.223.62:44335 requested a full tablet report, sending...
I20260806 15:58:17.522387 12192 ts_manager.cc:194] Registered new tserver with Master: 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 (127.13.223.1:37697)
I20260806 15:58:17.523531 12192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.1@KRBTEST.COM'} at 127.13.223.1:57551
WARNING: no policy specified for kudu/127.13.223.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
Entry for principal kudu/127.13.223.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab.
I20260806 15:58:17.545733 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44335
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:17.664083 12489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:17.664348 12489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:17.664402 12489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:17.668462 12489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:17.668601 12489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:17.668665 12489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:58:17.673977 12489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.2.keytab
--principal=kudu/127.13.223.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:17.675001 12489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:17.676023 12489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:17.685698 12497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.685716 12501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.685708 12496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:17.686125 12489 server_base.cc:1034] running on GCE node
I20260806 15:58:17.686524 12489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:17.687134 12489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:17.688315 12489 hybrid_clock.cc:648] HybridClock initialized: now 1786031897688281 us; error 46 us; skew 500 ppm
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:17.691087 12489 init.cc:377] Logged in from keytab as kudu/127.13.223.2@KRBTEST.COM (short username kudu)
I20260806 15:58:17.692021 12489 webserver.cc:460] Webserver started at http://127.13.223.2:43765/ using document root <none> and password file <none>
I20260806 15:58:17.692598 12489 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:17.692667 12489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:17.692835 12489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:17.694782 12489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "e8a31bc8eb534a18a2e479d722926d6a"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "0f2ca1db48a6db5e22a0b0dc8558771b"
server_key_iv: "0339e787b445aef4c5a4c622616574b3"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.695283 12489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "e8a31bc8eb534a18a2e479d722926d6a"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "0f2ca1db48a6db5e22a0b0dc8558771b"
server_key_iv: "0339e787b445aef4c5a4c622616574b3"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.699071 12489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:58:17.701578 12512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.702936 12489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:17.703095 12489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "e8a31bc8eb534a18a2e479d722926d6a"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "0f2ca1db48a6db5e22a0b0dc8558771b"
server_key_iv: "0339e787b445aef4c5a4c622616574b3"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.703219 12489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:17.729483 12489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:17.732973 12489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:17.733230 12489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:17.733937 12489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:17.734933 12489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:17.734989 12489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.735071 12489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:17.735116 12489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.746856 12489 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:38491
I20260806 15:58:17.746877 12688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:38491 every 8 connection(s)
I20260806 15:58:17.747880 12489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:58:17.753650 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 12489
I20260806 15:58:17.753805 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:58:17.754137 14204 external_mini_cluster.cc:1428] Setting key 25068bf1628cf174088a9af6af725d31
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.2@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:17.763586 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.749980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:38379 (local address 127.13.223.62:44335)
0806 15:58:17.750170 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:17.750177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:17.751063 (+ 886us) server_negotiation.cc:407] Connection header received
0806 15:58:17.752361 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:17.752364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:17.752412 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:17.752550 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:17.753994 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.754608 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.756048 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.756409 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.756606 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.759217 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:17.759224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:17.759229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:17.759262 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:17.761290 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.761841 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.761847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.761852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.761965 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.762352 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.762357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.762362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.762634 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:17.762751 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:17.763252 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 15:58:17.763380 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 15:58:17.764370 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.750403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44335 (local address 127.13.223.2:38379)
0806 15:58:17.750897 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:17.750943 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:17.752138 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:17.752723 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:17.752730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:17.753142 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:17.753783 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:17.753805 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.754735 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.754738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:17.755915 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:17.755925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.756606 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.756609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:17.756756 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.757375 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:17.757392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:17.759065 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:17.761411 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:17.761416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:17.761428 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:17.761723 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:17.762095 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:17.762098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:17.762099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:17.762247 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:17.762756 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:17.762762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:17.762991 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 15:58:17.763167 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 15:58:17.763437 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.13.223.3@KRBTEST.COM; defaulting to no policy
I20260806 15:58:17.765829 12689 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44335
I20260806 15:58:17.766106 12689 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:17.766726 12689 heartbeater.cc:511] Master 127.13.223.62:44335 requested a full tablet report, sending...
I20260806 15:58:17.767848 12192 ts_manager.cc:194] Registered new tserver with Master: e8a31bc8eb534a18a2e479d722926d6a (127.13.223.2:38491)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.3@KRBTEST.COM" created.
I20260806 15:58:17.768604 12192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.2@KRBTEST.COM'} at 127.13.223.2:38379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
Entry for principal kudu/127.13.223.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab.
I20260806 15:58:17.785190 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44335
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:17.899745 12698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:17.900003 12698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:17.900045 12698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:17.904059 12698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:17.904117 12698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:17.904173 12698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:58:17.909400 12698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.3.keytab
--principal=kudu/127.13.223.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:17.910435 12698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:17.911405 12698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:17.921029 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.921048 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:17.921046 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:17.921314 12698 server_base.cc:1034] running on GCE node
I20260806 15:58:17.921751 12698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:17.922395 12698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:17.923559 12698 hybrid_clock.cc:648] HybridClock initialized: now 1786031897923527 us; error 43 us; skew 500 ppm
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:17.926359 12698 init.cc:377] Logged in from keytab as kudu/127.13.223.3@KRBTEST.COM (short username kudu)
I20260806 15:58:17.927239 12698 webserver.cc:460] Webserver started at http://127.13.223.3:42657/ using document root <none> and password file <none>
I20260806 15:58:17.927763 12698 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:17.927822 12698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:17.927983 12698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:17.930047 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "686da707a933497ca4e11c531b00157a"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "feea0ae15f7801f70380ddf9b2c114eb"
server_key_iv: "d8d72d37e2d44b2cc2495a0ee8ca3ddb"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.930555 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "686da707a933497ca4e11c531b00157a"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "feea0ae15f7801f70380ddf9b2c114eb"
server_key_iv: "d8d72d37e2d44b2cc2495a0ee8ca3ddb"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.934448 12698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:17.937084 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.938470 12698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:17.938601 12698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "686da707a933497ca4e11c531b00157a"
format_stamp: "Formatted at 2026-08-06 15:58:17 on dist-test-slave-rwrg"
server_key: "feea0ae15f7801f70380ddf9b2c114eb"
server_key_iv: "d8d72d37e2d44b2cc2495a0ee8ca3ddb"
server_key_version: "encryptionkey@0"
I20260806 15:58:17.938725 12698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:17.959074 12698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:17.962642 12698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:17.962848 12698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:17.963470 12698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:17.964340 12698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:17.964409 12698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.964483 12698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:17.964558 12698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:17.974514 12698 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:33787
I20260806 15:58:17.974646 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:33787 every 8 connection(s)
I20260806 15:58:17.975515 12698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:58:17.983217 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 12698
I20260806 15:58:17.983392 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:58:17.983796 14204 external_mini_cluster.cc:1428] Setting key d4c020cb75522bdd29aaf7d398eb3ec1
Aug 06 15:58:17 dist-test-slave-rwrg krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031897, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.3@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:17.991217 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.977519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:35067 (local address 127.13.223.62:44335)
0806 15:58:17.977695 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:17.977700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:17.978849 (+ 1149us) server_negotiation.cc:407] Connection header received
0806 15:58:17.979777 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:17.979780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:17.979829 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:17.979897 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:17.981346 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.981877 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.983649 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.983894 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.984018 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.987366 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:17.987369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:17.987371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:17.987400 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:17.989120 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.989654 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.989657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.989658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.989746 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:17.990130 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:17.990133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:17.990135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:17.990333 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:17.990419 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:17.990973 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 15:58:17.991093 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 15:58:17.992214 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:17.977943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:44335 (local address 127.13.223.3:35067)
0806 15:58:17.978691 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:17.978737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:17.979570 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:17.980083 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:17.980091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:17.980482 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:17.981177 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:17.981194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.982049 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.982052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:17.983406 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:17.983424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:17.984114 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:17.984121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:17.984381 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:17.985331 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:17.985353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:17.987202 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:17.989235 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:17.989240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:17.989265 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:17.989539 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:17.989854 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:17.989856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:17.989857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:17.990014 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:17.990440 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:17.990446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:17.990733 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 15:58:17.990910 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 15:58:17.991206 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":113,"threads_started":1}
I20260806 15:58:17.993479 12908 heartbeater.cc:347] Connected to a master server at 127.13.223.62:44335
I20260806 15:58:17.993777 12908 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:17.994298 12908 heartbeater.cc:511] Master 127.13.223.62:44335 requested a full tablet report, sending...
I20260806 15:58:17.995319 12192 ts_manager.cc:194] Registered new tserver with Master: 686da707a933497ca4e11c531b00157a (127.13.223.3:33787)
I20260806 15:58:17.995895 12192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.3@KRBTEST.COM'} at 127.13.223.3:35067
I20260806 15:58:17.998339 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:58:18.007879 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.000294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.13.223.62:44335)
0806 15:58:18.000415 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.000421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.000629 (+ 208us) server_negotiation.cc:407] Connection header received
0806 15:58:18.000780 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.000783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.000830 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.000907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:18.001608 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.002112 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.002957 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.003187 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.003295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.004731 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:18.004735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:18.004737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:18.004765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:18.006153 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:18.006578 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:18.006581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:18.006583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:18.006664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:18.006930 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:18.006934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:18.006935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:18.007177 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:18.007314 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.007537 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.007696 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 15:58:18.010572 12192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:58:18.013496 12192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:58:18.028461 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.021984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.13.223.3:33787)
0806 15:58:18.023178 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.023190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.023212 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:58:18.023291 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.023296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.023502 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.023667 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:18.024060 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.025356 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.026963 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.027699 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.027812 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.027962 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.028170 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.028288 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":176,"threads_started":1}
I20260806 15:58:18.028461 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.021851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:33787 (local address 127.0.0.1:34508)
0806 15:58:18.022681 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.022733 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.022869 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.023669 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.023675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.023712 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:18.023882 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.023894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.025490 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.025494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.026826 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.026837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.027829 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.027836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.027947 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.027989 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.028131 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.028295 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":73,"spinlock_wait_cycles":12032,"thread_start_us":74,"threads_started":1}
I20260806 15:58:18.028883 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.022187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38491 (local address 127.0.0.1:37650)
0806 15:58:18.023089 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.023111 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.023191 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.023933 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.023936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.023952 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:18.024610 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.024621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.025666 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.025669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.027227 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.027239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.028366 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.028370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.028532 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.028562 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.028651 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.028726 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":730,"thread_start_us":78,"threads_started":1}
I20260806 15:58:18.029364 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.021920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:37697 (local address 127.0.0.1:35712)
0806 15:58:18.022716 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.022733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.022869 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.023673 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.023676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.023718 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:18.024072 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.024080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.025533 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.025537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.027227 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.027239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.028367 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.028371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.029056 (+ 685us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.029079 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.029146 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.029210 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":960,"spinlock_wait_cycles":14080,"thread_start_us":117,"threads_started":1}
I20260806 15:58:18.029873 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.022686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.13.223.2:38491)
0806 15:58:18.023467 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.023480 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.023502 (+ 22us) server_negotiation.cc:407] Connection header received
0806 15:58:18.023573 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.023577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.023781 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.023955 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:18.024728 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.025508 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.027381 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.028060 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.028158 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.028304 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.029646 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.029742 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":99,"threads_started":1}
I20260806 15:58:18.030601 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.022529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.13.223.1:37697)
0806 15:58:18.023178 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.023187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.023212 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:58:18.023291 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.023296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.023503 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.023896 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:18.024201 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.025356 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.027369 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.028213 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.028519 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.030312 (+ 1793us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.030366 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.030454 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":106,"threads_started":1}
I20260806 15:58:18.032236 12800 tablet_service.cc:1467] Processing CreateTablet for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f (DEFAULT_TABLE table=test-table [id=cb6490d4cebe405c984f8f72096c9e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:18.032528 12588 tablet_service.cc:1467] Processing CreateTablet for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f (DEFAULT_TABLE table=test-table [id=cb6490d4cebe405c984f8f72096c9e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:18.033263 12800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:18.033591 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:18.033252 12379 tablet_service.cc:1467] Processing CreateTablet for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f (DEFAULT_TABLE table=test-table [id=cb6490d4cebe405c984f8f72096c9e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:18.034219 12379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:18.039348 12935 tablet_bootstrap.cc:492] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Bootstrap starting.
I20260806 15:58:18.041337 12938 tablet_bootstrap.cc:492] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Bootstrap starting.
I20260806 15:58:18.041693 12936 tablet_bootstrap.cc:492] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Bootstrap starting.
I20260806 15:58:18.042687 12935 tablet_bootstrap.cc:654] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:18.043565 12938 tablet_bootstrap.cc:654] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:18.043941 12935 log.cc:824] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:18.044054 12936 tablet_bootstrap.cc:654] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:18.044607 12938 log.cc:824] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:18.044967 12936 log.cc:824] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:18.046540 12935 tablet_bootstrap.cc:492] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: No bootstrap required, opened a new log
I20260806 15:58:18.046540 12938 tablet_bootstrap.cc:492] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: No bootstrap required, opened a new log
I20260806 15:58:18.046751 12938 ts_tablet_manager.cc:1397] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:58:18.046751 12935 ts_tablet_manager.cc:1397] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 15:58:18.049940 12936 tablet_bootstrap.cc:492] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: No bootstrap required, opened a new log
I20260806 15:58:18.050180 12936 ts_tablet_manager.cc:1397] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260806 15:58:18.050973 12935 raft_consensus.cc:348] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.051586 12935 raft_consensus.cc:374] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:18.051645 12935 raft_consensus.cc:729] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686da707a933497ca4e11c531b00157a, State: Initialized, Role: FOLLOWER
I20260806 15:58:18.051520 12938 raft_consensus.cc:348] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.051787 12938 raft_consensus.cc:374] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:18.051842 12938 raft_consensus.cc:729] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae5ad3e0bb4463cbf6c0a320cd3c7d5, State: Initialized, Role: FOLLOWER
I20260806 15:58:18.052166 12935 consensus_queue.cc:260] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.052205 12938 consensus_queue.cc:260] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.053148 12908 heartbeater.cc:503] Master 127.13.223.62:44335 was elected leader, sending a full tablet report...
I20260806 15:58:18.053263 12938 ts_tablet_manager.cc:1428] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 15:58:18.053263 12935 ts_tablet_manager.cc:1428] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 15:58:18.053668 12483 heartbeater.cc:503] Master 127.13.223.62:44335 was elected leader, sending a full tablet report...
I20260806 15:58:18.054147 12936 raft_consensus.cc:348] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.054442 12936 raft_consensus.cc:374] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:18.054507 12936 raft_consensus.cc:729] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a31bc8eb534a18a2e479d722926d6a, State: Initialized, Role: FOLLOWER
I20260806 15:58:18.054970 12936 consensus_queue.cc:260] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.055634 12689 heartbeater.cc:503] Master 127.13.223.62:44335 was elected leader, sending a full tablet report...
I20260806 15:58:18.056033 12936 ts_tablet_manager.cc:1428] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 15:58:18.092463 12943 raft_consensus.cc:482] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:18.092762 12943 raft_consensus.cc:504] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.093935 12943 leader_election.cc:288] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 686da707a933497ca4e11c531b00157a (127.13.223.3:33787), e8a31bc8eb534a18a2e479d722926d6a (127.13.223.2:38491)
I20260806 15:58:18.099347 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.094262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:38491 (local address 127.13.223.1:57353)
0806 15:58:18.094736 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.094751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.094841 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.095081 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.095083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.095134 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:18.095252 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.095260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.096304 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.096308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.097895 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.097907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.098857 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.098860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.098957 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.098995 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.099080 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.099163 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":113,"spinlock_wait_cycles":768,"thread_start_us":122,"threads_started":1}
I20260806 15:58:18.099411 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.094476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:57353 (local address 127.13.223.2:38491)
0806 15:58:18.094610 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.094614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.094826 (+ 212us) server_negotiation.cc:407] Connection header received
0806 15:58:18.094921 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.094924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.094986 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.095069 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:18.095369 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.096179 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.098046 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.098726 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.098864 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.098945 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.099137 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.099244 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 15:58:18.099504 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.094304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:41011 (local address 127.13.223.3:33787)
0806 15:58:18.094559 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.094565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.094681 (+ 116us) server_negotiation.cc:407] Connection header received
0806 15:58:18.094848 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.094852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.094909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.094995 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:18.095369 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.096177 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.098046 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.098727 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.098864 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.098924 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.099281 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.099367 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260806 15:58:18.100039 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.094147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:33787 (local address 127.13.223.1:41011)
0806 15:58:18.094557 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.094572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.094711 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.095012 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.095015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.095044 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:18.095252 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.095260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.096304 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.096308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.097895 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.097907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.098997 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.099001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.099125 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.099150 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.099803 (+ 653us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.099895 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":111,"spinlock_wait_cycles":58496,"thread_start_us":151,"threads_started":1}
I20260806 15:58:18.100458 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6299a866ba1a4c6aa2fcff5b26e75c6f" candidate_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a31bc8eb534a18a2e479d722926d6a" is_pre_election: true
I20260806 15:58:18.100845 12611 raft_consensus.cc:2454] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 in term 0.
I20260806 15:58:18.101011 12836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6299a866ba1a4c6aa2fcff5b26e75c6f" candidate_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686da707a933497ca4e11c531b00157a" is_pre_election: true
I20260806 15:58:18.101320 12836 raft_consensus.cc:2454] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 in term 0.
I20260806 15:58:18.101433 12308 leader_election.cc:302] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae5ad3e0bb4463cbf6c0a320cd3c7d5, e8a31bc8eb534a18a2e479d722926d6a; no voters:
I20260806 15:58:18.101778 12943 raft_consensus.cc:2781] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:58:18.101863 12943 raft_consensus.cc:482] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:58:18.101907 12943 raft_consensus.cc:3037] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:18.102955 12943 raft_consensus.cc:504] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.103356 12943 leader_election.cc:288] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [CANDIDATE]: Term 1 election: Requested vote from peers 686da707a933497ca4e11c531b00157a (127.13.223.3:33787), e8a31bc8eb534a18a2e479d722926d6a (127.13.223.2:38491)
I20260806 15:58:18.103698 12836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6299a866ba1a4c6aa2fcff5b26e75c6f" candidate_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686da707a933497ca4e11c531b00157a"
I20260806 15:58:18.103819 12836 raft_consensus.cc:3037] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:18.105360 12836 raft_consensus.cc:2454] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 in term 1.
I20260806 15:58:18.105685 12310 leader_election.cc:302] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae5ad3e0bb4463cbf6c0a320cd3c7d5, 686da707a933497ca4e11c531b00157a; no voters:
I20260806 15:58:18.105722 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6299a866ba1a4c6aa2fcff5b26e75c6f" candidate_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a31bc8eb534a18a2e479d722926d6a"
I20260806 15:58:18.105834 12611 raft_consensus.cc:3037] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:18.107340 12611 raft_consensus.cc:2454] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 in term 1.
I20260806 15:58:18.107560 12943 raft_consensus.cc:2781] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:18.109577 12943 raft_consensus.cc:686] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5ae5ad3e0bb4463cbf6c0a320cd3c7d5, State: Running, Role: LEADER
I20260806 15:58:18.109956 12943 consensus_queue.cc:237] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } }
I20260806 15:58:18.113497 12192 catalog_manager.cc:5495] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 (127.13.223.1). New cstate: current_term: 1 leader_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ae5ad3e0bb4463cbf6c0a320cd3c7d5" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 } health_report { overall_health: UNKNOWN } } }
I20260806 15:58:18.134476 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.130541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.13.223.1:37697)
0806 15:58:18.130731 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.130737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.130754 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:58:18.130861 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.130865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.130932 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.131050 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:18.131326 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.131910 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.132921 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.133170 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.133283 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.133617 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:18.134058 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:18.134201 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.134255 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.134319 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 15:58:18.141340 12611 raft_consensus.cc:1261] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 1 FOLLOWER]: Refusing update from remote peer 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:18.141377 12836 raft_consensus.cc:1261] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 1 FOLLOWER]: Refusing update from remote peer 5ae5ad3e0bb4463cbf6c0a320cd3c7d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:18.142131 12954 consensus_queue.cc:1035] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686da707a933497ca4e11c531b00157a" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:18.142314 12943 consensus_queue.cc:1035] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a31bc8eb534a18a2e479d722926d6a" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:18.150415 12962 mvcc.cc:205] Tried to move back new op lower bound from 7315586654777860096 to 7315586654661349376. Current Snapshot: MvccSnapshot[applied={T|T < 7315586654777860096}]
I20260806 15:58:18.151415 12966 mvcc.cc:205] Tried to move back new op lower bound from 7315586654777860096 to 7315586654661349376. Current Snapshot: MvccSnapshot[applied={T|T < 7315586654777860096}]
I20260806 15:58:18.161031 12192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:18.161219 12192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:18.164044 12192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e615c8b04dad432882b1045c2b7825db: Sending DeleteTablet for 3 replicas of tablet 6299a866ba1a4c6aa2fcff5b26e75c6f
I20260806 15:58:18.164956 12379 tablet_service.cc:1514] Processing DeleteTablet for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:18 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:35712
I20260806 15:58:18.165046 12800 tablet_service.cc:1514] Processing DeleteTablet for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:18 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:34508
I20260806 15:58:18.165030 12588 tablet_service.cc:1514] Processing DeleteTablet for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:18 UTC) from {username='kudu', principal='kudu/127.13.223.62@KRBTEST.COM'} at 127.0.0.1:37650
I20260806 15:58:18.165598 12984 tablet_replica.cc:330] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: stopping tablet replica
I20260806 15:58:18.165834 12983 tablet_replica.cc:330] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: stopping tablet replica
I20260806 15:58:18.165984 12984 raft_consensus.cc:2229] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:18.166141 12983 raft_consensus.cc:2229] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:18.166348 12984 raft_consensus.cc:2258] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:18.166790 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 12289
I20260806 15:58:18.166395 12983 raft_consensus.cc:2258] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:18.180824 12984 ts_tablet_manager.cc:1905] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:18.181134 12983 ts_tablet_manager.cc:1905] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:18.183393 12983 ts_tablet_manager.cc:1918] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:18.183521 12983 log.cc:1196] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/6299a866ba1a4c6aa2fcff5b26e75c6f
I20260806 15:58:18.183840 12983 ts_tablet_manager.cc:1939] T 6299a866ba1a4c6aa2fcff5b26e75c6f P 686da707a933497ca4e11c531b00157a: Deleting consensus metadata
I20260806 15:58:18.185071 12984 ts_tablet_manager.cc:1918] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:18.185123 12171 catalog_manager.cc:4841] TS 686da707a933497ca4e11c531b00157a (127.13.223.3:33787): tablet 6299a866ba1a4c6aa2fcff5b26e75c6f (table test-table [id=cb6490d4cebe405c984f8f72096c9e53]) successfully deleted
I20260806 15:58:18.185207 12984 log.cc:1196] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/6299a866ba1a4c6aa2fcff5b26e75c6f
I20260806 15:58:18.185632 12984 ts_tablet_manager.cc:1939] T 6299a866ba1a4c6aa2fcff5b26e75c6f P e8a31bc8eb534a18a2e479d722926d6a: Deleting consensus metadata
W20260806 15:58:18.187191 12169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:37697 (error 108)
I20260806 15:58:18.187212 12167 catalog_manager.cc:4841] TS e8a31bc8eb534a18a2e479d722926d6a (127.13.223.2:38491): tablet 6299a866ba1a4c6aa2fcff5b26e75c6f (table test-table [id=cb6490d4cebe405c984f8f72096c9e53]) successfully deleted
I20260806 15:58:18.188215 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 12489
I20260806 15:58:18.188462 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.188077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:37697 (local address 127.0.0.1:35728)
0806 15:58:18.188242 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.188350 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 15:58:18.203217 12169 catalog_manager.cc:4584] TS 5ae5ad3e0bb4463cbf6c0a320cd3c7d5 (127.13.223.1:37697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6299a866ba1a4c6aa2fcff5b26e75c6f: Network error: Client connection negotiation failed: client connection to 127.13.223.1:37697: connect: Connection refused (error 111)
I20260806 15:58:18.208806 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 12698
I20260806 15:58:18.250309 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 12141
2026-08-06T15:58:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2604 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T15:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:18Z Disabled control of system clock
I20260806 15:58:18.288960 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:35777
--webserver_interface=127.13.223.62
--webserver_port=0
--builtin_ntp_servers=127.13.223.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 15:58:18.414110 12990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:18.414398 12990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:18.414438 12990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:18.419020 12990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:18.419088 12990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:18.419104 12990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:18.419121 12990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:18.419137 12990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:18.423902 12990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:35777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:18.425096 12990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:18.426469 12990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:18.434931 12996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:18.435165 12990 server_base.cc:1034] running on GCE node
W20260806 15:58:18.434924 12995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:18.434921 12998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:18.435855 12990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:18.437047 12990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:18.438236 12990 hybrid_clock.cc:648] HybridClock initialized: now 1786031898438221 us; error 29 us; skew 500 ppm
I20260806 15:58:18.447328 12990 webserver.cc:460] Webserver started at http://127.13.223.62:37515/ using document root <none> and password file <none>
I20260806 15:58:18.448012 12990 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:18.448113 12990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:18.448341 12990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:18.450551 12990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "b44ee42339ce4f2aac6020c3c6459667"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "6f28d63e228ce139bdffc1afce3e41d9"
server_key_iv: "fa5b1011159125183060d2ee32df5657"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.451184 12990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "b44ee42339ce4f2aac6020c3c6459667"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "6f28d63e228ce139bdffc1afce3e41d9"
server_key_iv: "fa5b1011159125183060d2ee32df5657"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.455022 12990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 15:58:18.457513 13010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.458842 12990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:18.458987 12990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "b44ee42339ce4f2aac6020c3c6459667"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "6f28d63e228ce139bdffc1afce3e41d9"
server_key_iv: "fa5b1011159125183060d2ee32df5657"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.459100 12990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:18.475445 12990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:18.476142 12990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:18.476342 12990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:18.483843 12990 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:35777
I20260806 15:58:18.483853 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:35777 every 8 connection(s)
I20260806 15:58:18.484833 12990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:18.485906 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 12990
I20260806 15:58:18.486042 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:18.486351 14204 external_mini_cluster.cc:1428] Setting key 4502fc1408a6cb1397d5eb85e4146bf3
I20260806 15:58:18.488642 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:18.494956 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667: Bootstrap starting.
I20260806 15:58:18.495815 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.488055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.13.223.62:35777)
0806 15:58:18.488745 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.488765 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.488811 (+ 46us) server_negotiation.cc:407] Connection header received
0806 15:58:18.489463 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.489480 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.489869 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.490158 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:18.490479 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.491449 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.492617 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.493082 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.493208 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.494080 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:18.494085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:18.494113 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:18.494485 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:18.494540 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:18.494559 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:18.494645 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.494801 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.495106 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":141,"threads_started":1}
I20260806 15:58:18.497402 13093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:18.498245 13093 log.cc:824] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:18.500090 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667: No bootstrap required, opened a new log
I20260806 15:58:18.503599 13093 raft_consensus.cc:348] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ee42339ce4f2aac6020c3c6459667" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 35777 } }
I20260806 15:58:18.503849 13093 raft_consensus.cc:374] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:18.503924 13093 raft_consensus.cc:729] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44ee42339ce4f2aac6020c3c6459667, State: Initialized, Role: FOLLOWER
I20260806 15:58:18.504333 13093 consensus_queue.cc:260] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ee42339ce4f2aac6020c3c6459667" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 35777 } }
I20260806 15:58:18.504477 13093 raft_consensus.cc:388] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:18.504617 13093 raft_consensus.cc:482] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:18.504717 13093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:18.505882 13093 raft_consensus.cc:504] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ee42339ce4f2aac6020c3c6459667" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 35777 } }
I20260806 15:58:18.506232 13093 leader_election.cc:302] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b44ee42339ce4f2aac6020c3c6459667; no voters:
I20260806 15:58:18.506562 13093 leader_election.cc:288] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:18.506817 13100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:18.507050 13100 raft_consensus.cc:686] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [term 1 LEADER]: Becoming Leader. State: Replica: b44ee42339ce4f2aac6020c3c6459667, State: Running, Role: LEADER
I20260806 15:58:18.507356 13100 consensus_queue.cc:237] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ee42339ce4f2aac6020c3c6459667" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 35777 } }
I20260806 15:58:18.508164 13093 sys_catalog.cc:564] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:18.509213 13101 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b44ee42339ce4f2aac6020c3c6459667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ee42339ce4f2aac6020c3c6459667" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 35777 } } }
I20260806 15:58:18.509361 13101 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:18.509722 13102 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b44ee42339ce4f2aac6020c3c6459667. Latest consensus state: current_term: 1 leader_uuid: "b44ee42339ce4f2aac6020c3c6459667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ee42339ce4f2aac6020c3c6459667" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 35777 } } }
I20260806 15:58:18.509814 13102 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:18.510416 13114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:18.513100 13114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:18.518889 13114 catalog_manager.cc:1347] Generated new cluster ID: 6a72cc72231e418dab0f6684ec2daec2
I20260806 15:58:18.518954 13114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:18.534819 13114 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:18.536094 13114 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:18.541641 13114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667: Generated new TSK 0
I20260806 15:58:18.542412 13114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:18.547178 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.1:0
--local_ip_for_outbound_sockets=127.13.223.1
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:35777
--builtin_ntp_servers=127.13.223.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 15:58:18.665156 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:18.665449 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:18.665489 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:18.669586 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:18.669653 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:18.669714 13128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.1
I20260806 15:58:18.674500 13128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.1
--webserver_port=0
--tserver_master_addrs=127.13.223.62:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.1
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:18.675609 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:18.677016 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:18.687017 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:18.687026 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:18.687124 13128 server_base.cc:1034] running on GCE node
W20260806 15:58:18.687026 13139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:18.687716 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:18.688369 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:18.689555 13128 hybrid_clock.cc:648] HybridClock initialized: now 1786031898689523 us; error 43 us; skew 500 ppm
I20260806 15:58:18.691563 13128 webserver.cc:460] Webserver started at http://127.13.223.1:39473/ using document root <none> and password file <none>
I20260806 15:58:18.692163 13128 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:18.692258 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:18.692528 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:18.694725 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data/instance:
uuid: "703d97b987ec48e79eebbe671c52e7bd"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "318b7cf089353e052af93b621d5c2ae5"
server_key_iv: "a246876b6e11e125a624949de5a0caf5"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.695308 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance:
uuid: "703d97b987ec48e79eebbe671c52e7bd"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "318b7cf089353e052af93b621d5c2ae5"
server_key_iv: "a246876b6e11e125a624949de5a0caf5"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.699486 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:58:18.702544 13149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.703954 13128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:58:18.704113 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal
uuid: "703d97b987ec48e79eebbe671c52e7bd"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "318b7cf089353e052af93b621d5c2ae5"
server_key_iv: "a246876b6e11e125a624949de5a0caf5"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.704252 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:18.722887 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:18.723685 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:18.723902 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:18.724601 13128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:18.725643 13128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:18.725719 13128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.725798 13128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:18.725850 13128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.735896 13128 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.1:42477
I20260806 15:58:18.735920 13322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.1:42477 every 8 connection(s)
I20260806 15:58:18.736922 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/data/info.pb
I20260806 15:58:18.745281 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 13128
I20260806 15:58:18.745456 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-0/wal/instance
I20260806 15:58:18.745788 14204 external_mini_cluster.cc:1428] Setting key 1ba156daa31f142f00d31148377600cf
I20260806 15:58:18.747344 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.738855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.1:37313 (local address 127.13.223.62:35777)
0806 15:58:18.739027 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.739033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.740182 (+ 1149us) server_negotiation.cc:407] Connection header received
0806 15:58:18.741790 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.741793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.741841 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.741920 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:18.742950 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.743492 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.744884 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.745231 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.745357 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.746765 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:18.746770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:18.746773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:18.746788 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:18.746817 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:18.746827 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:18.746915 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.747090 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.747172 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 15:58:18.747742 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.2:0
--local_ip_for_outbound_sockets=127.13.223.2
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:35777
--builtin_ntp_servers=127.13.223.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 15:58:18.748319 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.739277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:35777 (local address 127.13.223.1:37313)
0806 15:58:18.740014 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.740066 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.741583 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.742100 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.742107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.742196 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:18.742794 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.742809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.743781 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.743784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.744765 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.744776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.745465 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.745470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.745718 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.746463 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:18.746465 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:18.746516 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:18.746587 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:18.746919 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:18.746932 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:18.746941 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.747088 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.747384 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":114,"threads_started":1}
I20260806 15:58:18.749766 13323 heartbeater.cc:347] Connected to a master server at 127.13.223.62:35777
I20260806 15:58:18.750057 13323 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:18.750761 13323 heartbeater.cc:511] Master 127.13.223.62:35777 requested a full tablet report, sending...
I20260806 15:58:18.752777 13039 ts_manager.cc:194] Registered new tserver with Master: 703d97b987ec48e79eebbe671c52e7bd (127.13.223.1:42477)
I20260806 15:58:18.754755 13039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.1:37313
W20260806 15:58:18.869639 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:18.869896 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:18.869935 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:18.873922 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:18.873977 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:18.874029 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.2
I20260806 15:58:18.879024 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.2
--webserver_port=0
--tserver_master_addrs=127.13.223.62:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.2
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:18.880087 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:18.881405 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:18.891127 13339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:18.891112 13341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:18.891099 13338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:18.891288 13331 server_base.cc:1034] running on GCE node
I20260806 15:58:18.891819 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:18.892444 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:18.893658 13331 hybrid_clock.cc:648] HybridClock initialized: now 1786031898893627 us; error 43 us; skew 500 ppm
I20260806 15:58:18.895512 13331 webserver.cc:460] Webserver started at http://127.13.223.2:44323/ using document root <none> and password file <none>
I20260806 15:58:18.896106 13331 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:18.896170 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:18.896416 13331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:18.898479 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data/instance:
uuid: "9746ef36ad0145bd8408527529dd9d8e"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "4ccadd2bd04a10726c6e05d9146192db"
server_key_iv: "28265cbba902670f531e5e929d7fa8b4"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.899068 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance:
uuid: "9746ef36ad0145bd8408527529dd9d8e"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "4ccadd2bd04a10726c6e05d9146192db"
server_key_iv: "28265cbba902670f531e5e929d7fa8b4"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.902844 13331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:58:18.905694 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.906942 13331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:18.907083 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal
uuid: "9746ef36ad0145bd8408527529dd9d8e"
format_stamp: "Formatted at 2026-08-06 15:58:18 on dist-test-slave-rwrg"
server_key: "4ccadd2bd04a10726c6e05d9146192db"
server_key_iv: "28265cbba902670f531e5e929d7fa8b4"
server_key_version: "encryptionkey@0"
I20260806 15:58:18.907205 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:18.940433 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:18.941265 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:18.941489 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:18.942137 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:18.943123 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:18.943192 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.943286 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:18.943332 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:18.952700 13331 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.2:46489
I20260806 15:58:18.952766 13526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.2:46489 every 8 connection(s)
I20260806 15:58:18.953740 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/data/info.pb
I20260806 15:58:18.956866 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 13331
I20260806 15:58:18.957007 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal/instance
I20260806 15:58:18.957307 14204 external_mini_cluster.cc:1428] Setting key 66e0f701fa603a5846442ff33e4bb8f1
I20260806 15:58:18.959172 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.3:0
--local_ip_for_outbound_sockets=127.13.223.3
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:35777
--builtin_ntp_servers=127.13.223.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 15:58:18.963423 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.955649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.2:49941 (local address 127.13.223.62:35777)
0806 15:58:18.955792 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:18.955795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:18.956939 (+ 1144us) server_negotiation.cc:407] Connection header received
0806 15:58:18.958472 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:18.958476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:18.958530 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:18.958637 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:18.959586 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.960099 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.961317 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.961539 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.961656 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.962809 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:18.962816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:18.962818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:18.962830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:18.962849 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:18.962856 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:18.962930 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:18.963124 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 15:58:18.963205 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 15:58:18.964179 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:18.956049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:35777 (local address 127.13.223.2:49941)
0806 15:58:18.956793 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:18.956835 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:18.958266 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:18.958804 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:18.958811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:18.958896 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:18.959453 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.959468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.960224 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.960227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.961193 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:18.961203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:18.961695 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:18.961699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:18.961859 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:18.962552 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:18.962554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:18.962595 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:18.962662 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:18.962945 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:18.962960 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:18.962964 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 15:58:18.963117 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 15:58:18.963343 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260806 15:58:18.965551 13527 heartbeater.cc:347] Connected to a master server at 127.13.223.62:35777
I20260806 15:58:18.965801 13527 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:18.966360 13527 heartbeater.cc:511] Master 127.13.223.62:35777 requested a full tablet report, sending...
I20260806 15:58:18.967473 13039 ts_manager.cc:194] Registered new tserver with Master: 9746ef36ad0145bd8408527529dd9d8e (127.13.223.2:46489)
I20260806 15:58:18.968042 13039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.2:49941
W20260806 15:58:19.077483 13530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:19.077752 13530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:19.077819 13530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:19.081851 13530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:19.081908 13530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:19.081993 13530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.3
I20260806 15:58:19.086849 13530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.3
--webserver_port=0
--tserver_master_addrs=127.13.223.62:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.3
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:19.088025 13530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:19.089430 13530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:19.099097 13543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:19.099085 13539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:19.099092 13537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:19.099319 13530 server_base.cc:1034] running on GCE node
I20260806 15:58:19.099807 13530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:19.100445 13530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:19.101672 13530 hybrid_clock.cc:648] HybridClock initialized: now 1786031899101652 us; error 40 us; skew 500 ppm
I20260806 15:58:19.103526 13530 webserver.cc:460] Webserver started at http://127.13.223.3:37389/ using document root <none> and password file <none>
I20260806 15:58:19.104101 13530 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:19.104189 13530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:19.104410 13530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:19.106441 13530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data/instance:
uuid: "517a345719e8429eb28658c763f44302"
format_stamp: "Formatted at 2026-08-06 15:58:19 on dist-test-slave-rwrg"
server_key: "f796a85cbcb0c10e8f024f30c8518db2"
server_key_iv: "44c0c9f3ae8458d07b1e006bbc681855"
server_key_version: "encryptionkey@0"
I20260806 15:58:19.106989 13530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance:
uuid: "517a345719e8429eb28658c763f44302"
format_stamp: "Formatted at 2026-08-06 15:58:19 on dist-test-slave-rwrg"
server_key: "f796a85cbcb0c10e8f024f30c8518db2"
server_key_iv: "44c0c9f3ae8458d07b1e006bbc681855"
server_key_version: "encryptionkey@0"
I20260806 15:58:19.111076 13530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:19.113507 13550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:19.114671 13530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:19.114809 13530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal
uuid: "517a345719e8429eb28658c763f44302"
format_stamp: "Formatted at 2026-08-06 15:58:19 on dist-test-slave-rwrg"
server_key: "f796a85cbcb0c10e8f024f30c8518db2"
server_key_iv: "44c0c9f3ae8458d07b1e006bbc681855"
server_key_version: "encryptionkey@0"
I20260806 15:58:19.114914 13530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:19.146041 13530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:19.146800 13530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:19.147008 13530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:19.147615 13530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 15:58:19.148584 13530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 15:58:19.148653 13530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:19.148725 13530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 15:58:19.148777 13530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:19.158355 13530 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.3:35965
I20260806 15:58:19.158385 13735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.3:35965 every 8 connection(s)
I20260806 15:58:19.159281 13530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/data/info.pb
I20260806 15:58:19.167306 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 13530
I20260806 15:58:19.167456 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal/instance
I20260806 15:58:19.167743 14204 external_mini_cluster.cc:1428] Setting key ddbc8276969aeb24a528651ae27ba798
I20260806 15:58:19.169176 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.161347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:37571 (local address 127.13.223.62:35777)
0806 15:58:19.161538 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.161544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.162497 (+ 953us) server_negotiation.cc:407] Connection header received
0806 15:58:19.164144 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.164148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.164211 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.164295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:19.165349 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.165892 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.167083 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.167309 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.167423 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.168670 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:19.168674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:19.168677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:19.168690 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:19.168717 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:19.168725 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:19.168799 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.168963 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.169036 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 15:58:19.170068 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.161585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:35777 (local address 127.13.223.3:37571)
0806 15:58:19.162345 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.162388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:19.163928 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:19.164467 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:19.164474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:19.164641 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:19.165200 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.165213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.166032 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.166036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.166925 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.166934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.167473 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.167477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.167670 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.168356 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 15:58:19.168357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:19.168399 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 15:58:19.168463 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:19.168814 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:19.168824 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:19.168830 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 15:58:19.168962 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 15:58:19.169280 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260806 15:58:19.171180 13737 heartbeater.cc:347] Connected to a master server at 127.13.223.62:35777
I20260806 15:58:19.171386 13737 heartbeater.cc:464] Registering TS with master...
I20260806 15:58:19.171906 13737 heartbeater.cc:511] Master 127.13.223.62:35777 requested a full tablet report, sending...
I20260806 15:58:19.172895 13039 ts_manager.cc:194] Registered new tserver with Master: 517a345719e8429eb28658c763f44302 (127.13.223.3:35965)
I20260806 15:58:19.173413 13039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.3:37571
I20260806 15:58:19.181541 14204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 15:58:19.187012 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.183475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.13.223.62:35777)
0806 15:58:19.183643 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.183648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.183817 (+ 169us) server_negotiation.cc:407] Connection header received
0806 15:58:19.183870 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.183872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.183913 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.183986 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:19.184231 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.184782 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.185606 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.185809 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.185890 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.186629 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:19.186632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:19.186634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:19.186644 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:19.186661 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:19.186668 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:19.186727 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.186801 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.186843 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 15:58:19.189955 13039 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 15:58:19.192896 13039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 15:58:19.207121 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.201116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:46489 (local address 127.0.0.1:58294)
0806 15:58:19.201464 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.201517 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:19.201664 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:19.202455 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:19.202463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:19.202488 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:19.202643 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.202652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.204094 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.204097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.205563 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.205572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.206560 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.206563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.206685 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.206726 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 15:58:19.206839 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 15:58:19.206956 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":51,"thread_start_us":83,"threads_started":1}
I20260806 15:58:19.207170 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.201254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.13.223.2:46489)
0806 15:58:19.201738 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.201752 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.201777 (+ 25us) server_negotiation.cc:407] Connection header received
0806 15:58:19.201859 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.201864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.202072 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.202268 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:19.202781 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.203972 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.205681 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.206444 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.206545 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.206701 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.206884 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.206990 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":123,"threads_started":1}
I20260806 15:58:19.207144 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.201620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42477 (local address 127.0.0.1:42100)
0806 15:58:19.202521 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.202534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:19.202726 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:19.203194 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:19.203197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:19.203213 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:19.203323 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.203332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.204526 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.204530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.205389 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.205399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.206732 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.206736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.206835 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.206861 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 15:58:19.206934 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 15:58:19.207000 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":105,"threads_started":1}
I20260806 15:58:19.207500 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.202023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.13.223.1:42477)
0806 15:58:19.202392 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.202400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.202653 (+ 253us) server_negotiation.cc:407] Connection header received
0806 15:58:19.202876 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.202881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.203056 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.203203 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:19.203456 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.204396 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.205516 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.206329 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.206440 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.206659 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.207217 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.207327 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":74,"threads_started":1}
I20260806 15:58:19.208631 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.201856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.3:35965 (local address 127.0.0.1:37532)
0806 15:58:19.202809 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.202825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:19.202947 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:19.203466 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:19.203468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:19.203484 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:19.203572 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.203576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.205474 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.205477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.206291 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.206300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.207887 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.207891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.208238 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.208263 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:19.208333 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 15:58:19.208400 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":13,"thread_start_us":92,"threads_started":1}
I20260806 15:58:19.209242 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.202269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.13.223.3:35965)
0806 15:58:19.202665 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.202679 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.202927 (+ 248us) server_negotiation.cc:407] Connection header received
0806 15:58:19.203061 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.203065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.203233 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.203378 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:19.203677 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.204684 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.206410 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.207209 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.207893 (+ 684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.208088 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.209052 (+ 964us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.209126 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":89,"threads_started":1}
I20260806 15:58:19.210549 13425 tablet_service.cc:1467] Processing CreateTablet for tablet a26747700fae4ce181f682e1eca89ca1 (DEFAULT_TABLE table=test-table [id=5c4a0f3c53434354a7d6549363b7454c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:19.210549 13219 tablet_service.cc:1467] Processing CreateTablet for tablet a26747700fae4ce181f682e1eca89ca1 (DEFAULT_TABLE table=test-table [id=5c4a0f3c53434354a7d6549363b7454c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:19.211721 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26747700fae4ce181f682e1eca89ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:19.211721 13219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26747700fae4ce181f682e1eca89ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:19.211617 13642 tablet_service.cc:1467] Processing CreateTablet for tablet a26747700fae4ce181f682e1eca89ca1 (DEFAULT_TABLE table=test-table [id=5c4a0f3c53434354a7d6549363b7454c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 15:58:19.212711 13642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26747700fae4ce181f682e1eca89ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:19.218560 13771 tablet_bootstrap.cc:492] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Bootstrap starting.
I20260806 15:58:19.219754 13772 tablet_bootstrap.cc:492] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Bootstrap starting.
I20260806 15:58:19.220016 13773 tablet_bootstrap.cc:492] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: Bootstrap starting.
I20260806 15:58:19.220932 13771 tablet_bootstrap.cc:654] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:19.221864 13771 log.cc:824] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:19.221988 13773 tablet_bootstrap.cc:654] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:19.221989 13772 tablet_bootstrap.cc:654] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:19.222828 13772 log.cc:824] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:19.222828 13773 log.cc:824] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:19.223845 13771 tablet_bootstrap.cc:492] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: No bootstrap required, opened a new log
I20260806 15:58:19.224046 13771 ts_tablet_manager.cc:1397] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 15:58:19.224439 13772 tablet_bootstrap.cc:492] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: No bootstrap required, opened a new log
I20260806 15:58:19.224669 13772 ts_tablet_manager.cc:1397] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260806 15:58:19.225807 13773 tablet_bootstrap.cc:492] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: No bootstrap required, opened a new log
I20260806 15:58:19.225989 13773 ts_tablet_manager.cc:1397] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260806 15:58:19.227802 13771 raft_consensus.cc:348] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.228032 13771 raft_consensus.cc:374] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:19.228087 13771 raft_consensus.cc:729] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517a345719e8429eb28658c763f44302, State: Initialized, Role: FOLLOWER
I20260806 15:58:19.228564 13771 consensus_queue.cc:260] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.228951 13772 raft_consensus.cc:348] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.229202 13772 raft_consensus.cc:374] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:19.229282 13772 raft_consensus.cc:729] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9746ef36ad0145bd8408527529dd9d8e, State: Initialized, Role: FOLLOWER
I20260806 15:58:19.229571 13737 heartbeater.cc:503] Master 127.13.223.62:35777 was elected leader, sending a full tablet report...
I20260806 15:58:19.229820 13771 ts_tablet_manager.cc:1428] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260806 15:58:19.229753 13772 consensus_queue.cc:260] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.230245 13773 raft_consensus.cc:348] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.230458 13527 heartbeater.cc:503] Master 127.13.223.62:35777 was elected leader, sending a full tablet report...
I20260806 15:58:19.230486 13773 raft_consensus.cc:374] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:19.230536 13773 raft_consensus.cc:729] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703d97b987ec48e79eebbe671c52e7bd, State: Initialized, Role: FOLLOWER
I20260806 15:58:19.230928 13773 consensus_queue.cc:260] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.231591 13323 heartbeater.cc:503] Master 127.13.223.62:35777 was elected leader, sending a full tablet report...
I20260806 15:58:19.231999 13772 ts_tablet_manager.cc:1428] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 15:58:19.232967 13773 ts_tablet_manager.cc:1428] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 15:58:19.238112 13324 tablet.cc:2369] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 15:58:19.263906 13779 raft_consensus.cc:482] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 15:58:19.264165 13779 raft_consensus.cc:504] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.265491 13779 leader_election.cc:288] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9746ef36ad0145bd8408527529dd9d8e (127.13.223.2:46489), 703d97b987ec48e79eebbe671c52e7bd (127.13.223.1:42477)
I20260806 15:58:19.269294 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.265682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.2:46489 (local address 127.13.223.3:49193)
0806 15:58:19.265892 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.265909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:19.265998 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:19.266353 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:19.266357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:19.266376 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:19.266497 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.266503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.267276 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.267281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.268127 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.268136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.268834 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.268838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.268940 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.268968 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 15:58:19.269055 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 15:58:19.269129 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":51}
I20260806 15:58:19.269294 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.265970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:49193 (local address 127.13.223.2:46489)
0806 15:58:19.266095 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.266112 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.266126 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:19.266182 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.266185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.266238 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.266337 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:19.266616 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.267161 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.268259 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.268711 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.268839 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.268891 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.269073 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.269152 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 15:58:19.269327 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.266193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.3:36239 (local address 127.13.223.1:42477)
0806 15:58:19.266330 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.266336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.266351 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:58:19.266405 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.266406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.266443 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.266523 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 15:58:19.266799 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.267292 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.268300 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.268763 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.268890 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.268938 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.269126 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.269194 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 15:58:19.269340 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.265729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42477 (local address 127.13.223.3:36239)
0806 15:58:19.266169 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.266182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:19.266275 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:19.266543 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:19.266546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:19.266562 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 15:58:19.266674 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.266679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.267407 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.267410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.268179 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:19.268187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.268890 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.268894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:19.268996 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.269021 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 15:58:19.269113 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 15:58:19.269179 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":141,"threads_started":1}
I20260806 15:58:19.269971 13247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26747700fae4ce181f682e1eca89ca1" candidate_uuid: "517a345719e8429eb28658c763f44302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703d97b987ec48e79eebbe671c52e7bd" is_pre_election: true
I20260806 15:58:19.270015 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26747700fae4ce181f682e1eca89ca1" candidate_uuid: "517a345719e8429eb28658c763f44302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9746ef36ad0145bd8408527529dd9d8e" is_pre_election: true
I20260806 15:58:19.270284 13461 raft_consensus.cc:2454] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 517a345719e8429eb28658c763f44302 in term 0.
I20260806 15:58:19.270294 13247 raft_consensus.cc:2454] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 517a345719e8429eb28658c763f44302 in term 0.
I20260806 15:58:19.270790 13553 leader_election.cc:302] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 517a345719e8429eb28658c763f44302, 703d97b987ec48e79eebbe671c52e7bd, 9746ef36ad0145bd8408527529dd9d8e; no voters:
I20260806 15:58:19.271082 13779 raft_consensus.cc:2781] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 15:58:19.271170 13779 raft_consensus.cc:482] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 15:58:19.271214 13779 raft_consensus.cc:3037] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:19.272217 13779 raft_consensus.cc:504] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.272559 13779 leader_election.cc:288] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [CANDIDATE]: Term 1 election: Requested vote from peers 9746ef36ad0145bd8408527529dd9d8e (127.13.223.2:46489), 703d97b987ec48e79eebbe671c52e7bd (127.13.223.1:42477)
I20260806 15:58:19.272892 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26747700fae4ce181f682e1eca89ca1" candidate_uuid: "517a345719e8429eb28658c763f44302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9746ef36ad0145bd8408527529dd9d8e"
I20260806 15:58:19.272904 13247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26747700fae4ce181f682e1eca89ca1" candidate_uuid: "517a345719e8429eb28658c763f44302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703d97b987ec48e79eebbe671c52e7bd"
I20260806 15:58:19.273025 13461 raft_consensus.cc:3037] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:19.273041 13247 raft_consensus.cc:3037] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:19.274046 13461 raft_consensus.cc:2454] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517a345719e8429eb28658c763f44302 in term 1.
I20260806 15:58:19.274159 13247 raft_consensus.cc:2454] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517a345719e8429eb28658c763f44302 in term 1.
I20260806 15:58:19.274376 13551 leader_election.cc:302] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 517a345719e8429eb28658c763f44302, 9746ef36ad0145bd8408527529dd9d8e; no voters:
I20260806 15:58:19.274555 13779 raft_consensus.cc:2781] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:19.274776 13779 raft_consensus.cc:686] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 1 LEADER]: Becoming Leader. State: Replica: 517a345719e8429eb28658c763f44302, State: Running, Role: LEADER
I20260806 15:58:19.275134 13779 consensus_queue.cc:237] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } }
I20260806 15:58:19.278373 13039 catalog_manager.cc:5495] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 517a345719e8429eb28658c763f44302 (127.13.223.3). New cstate: current_term: 1 leader_uuid: "517a345719e8429eb28658c763f44302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "517a345719e8429eb28658c763f44302" member_type: VOTER last_known_addr { host: "127.13.223.3" port: 35965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 } health_report { overall_health: UNKNOWN } } }
I20260806 15:58:19.289345 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.286079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.13.223.3:35965)
0806 15:58:19.286192 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:19.286195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:19.286207 (+ 12us) server_negotiation.cc:407] Connection header received
0806 15:58:19.286341 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:19.286344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:19.286392 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:19.286475 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:19.286682 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.287154 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.287975 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:19.288195 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:19.288270 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:19.288586 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:19.288987 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:19.289075 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:19.289158 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 15:58:19.289200 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 15:58:19.295194 13461 raft_consensus.cc:1261] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 1 FOLLOWER]: Refusing update from remote peer 517a345719e8429eb28658c763f44302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:19.295193 13247 raft_consensus.cc:1261] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 1 FOLLOWER]: Refusing update from remote peer 517a345719e8429eb28658c763f44302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:19.295992 13779 consensus_queue.cc:1035] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703d97b987ec48e79eebbe671c52e7bd" member_type: VOTER last_known_addr { host: "127.13.223.1" port: 42477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:19.296226 13788 consensus_queue.cc:1035] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9746ef36ad0145bd8408527529dd9d8e" member_type: VOTER last_known_addr { host: "127.13.223.2" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:19.305114 13794 mvcc.cc:205] Tried to move back new op lower bound from 7315586659505864704 to 7315586659434033152. Current Snapshot: MvccSnapshot[applied={T|T < 7315586659505864704}]
I20260806 15:58:19.314623 13039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:19.314769 13039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 15:58:19.317180 13039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b44ee42339ce4f2aac6020c3c6459667: Sending DeleteTablet for 3 replicas of tablet a26747700fae4ce181f682e1eca89ca1
I20260806 15:58:19.318084 13425 tablet_service.cc:1514] Processing DeleteTablet for tablet a26747700fae4ce181f682e1eca89ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:19 UTC) from {username='slave'} at 127.0.0.1:58294
I20260806 15:58:19.318078 13219 tablet_service.cc:1514] Processing DeleteTablet for tablet a26747700fae4ce181f682e1eca89ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:19 UTC) from {username='slave'} at 127.0.0.1:42100
I20260806 15:58:19.318640 13809 tablet_replica.cc:330] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: stopping tablet replica
I20260806 15:58:19.318677 13642 tablet_service.cc:1514] Processing DeleteTablet for tablet a26747700fae4ce181f682e1eca89ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 15:58:19 UTC) from {username='slave'} at 127.0.0.1:37532
I20260806 15:58:19.318989 13809 raft_consensus.cc:2229] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:19.319168 13811 tablet_replica.cc:330] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: stopping tablet replica
I20260806 15:58:19.319391 13809 raft_consensus.cc:2258] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:19.319542 13811 raft_consensus.cc:2229] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 1 LEADER]: Raft consensus shutting down.
I20260806 15:58:19.319777 13807 tablet_replica.cc:330] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd: stopping tablet replica
I20260806 15:58:19.319818 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 13128
I20260806 15:58:19.320124 13807 raft_consensus.cc:2229] T a26747700fae4ce181f682e1eca89ca1 P 703d97b987ec48e79eebbe671c52e7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 15:58:19.325142 13809 ts_tablet_manager.cc:1905] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:19.325649 13811 raft_consensus.cc:2258] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 15:58:19.328830 13809 ts_tablet_manager.cc:1918] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:19.329015 13809 log.cc:1196] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-1/wal/wals/a26747700fae4ce181f682e1eca89ca1
I20260806 15:58:19.329418 13809 ts_tablet_manager.cc:1939] T a26747700fae4ce181f682e1eca89ca1 P 9746ef36ad0145bd8408527529dd9d8e: Deleting consensus metadata
I20260806 15:58:19.334159 13011 catalog_manager.cc:4841] TS 9746ef36ad0145bd8408527529dd9d8e (127.13.223.2:46489): tablet a26747700fae4ce181f682e1eca89ca1 (table test-table [id=5c4a0f3c53434354a7d6549363b7454c]) successfully deleted
I20260806 15:58:19.335971 13811 ts_tablet_manager.cc:1905] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 15:58:19.338310 13811 ts_tablet_manager.cc:1918] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 15:58:19.338433 13811 log.cc:1196] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Deleting WAL directory at /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786031632208048-14204-0/minicluster-data/ts-2/wal/wals/a26747700fae4ce181f682e1eca89ca1
I20260806 15:58:19.338730 13811 ts_tablet_manager.cc:1939] T a26747700fae4ce181f682e1eca89ca1 P 517a345719e8429eb28658c763f44302: Deleting consensus metadata
I20260806 15:58:19.339852 13011 catalog_manager.cc:4841] TS 517a345719e8429eb28658c763f44302 (127.13.223.3:35965): tablet a26747700fae4ce181f682e1eca89ca1 (table test-table [id=5c4a0f3c53434354a7d6549363b7454c]) successfully deleted
W20260806 15:58:19.340185 13013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.1:42477 (error 108)
I20260806 15:58:19.341076 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 13331
I20260806 15:58:19.363080 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:19.362393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.1:42477 (local address 127.0.0.1:42104)
0806 15:58:19.362774 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:19.362925 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
W20260806 15:58:19.363376 13013 catalog_manager.cc:4584] TS 703d97b987ec48e79eebbe671c52e7bd (127.13.223.1:42477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a26747700fae4ce181f682e1eca89ca1: Network error: Client connection negotiation failed: client connection to 127.13.223.1:42477: connect: Connection refused (error 111)
I20260806 15:58:19.370417 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 13530
I20260806 15:58:19.385941 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 12990
2026-08-06T15:58:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1134 ms)
[----------] 6 tests from EncryptionPolicyTest (11508 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:19 dist-test-slave-rwrg krb5kdc[13816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:19 dist-test-slave-rwrg krb5kdc[13816](info): set up 2 sockets
Aug 06 15:58:19 dist-test-slave-rwrg krb5kdc[13816](info): commencing operation
krb5kdc: starting...
W20260806 15:58:20.658985 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:20 dist-test-slave-rwrg krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:20.791720 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:20.906723 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:20.907034 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:20.907109 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:20.911101 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:20.911188 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:20.911229 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:20.911285 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:20.911324 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:20.916358 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:20.917495 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:20.918586 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:20.927146 13841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:20.927168 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:20.927284 13832 server_base.cc:1034] running on GCE node
W20260806 15:58:20.927438 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:20.928072 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:20.929184 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:20.930334 13832 hybrid_clock.cc:648] HybridClock initialized: now 1786031900930292 us; error 58 us; skew 500 ppm
Aug 06 15:58:20 dist-test-slave-rwrg krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:20.933187 13832 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:20.934049 13832 webserver.cc:460] Webserver started at http://127.0.0.1:36091/ using document root <none> and password file <none>
I20260806 15:58:20.934594 13832 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:20.934665 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:20.934898 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:20.936905 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "759b693e0a89446cbffd7cfd89ab0881"
format_stamp: "Formatted at 2026-08-06 15:58:20 on dist-test-slave-rwrg"
server_key: "18ae30a23b4955957cab872cf73af7af"
server_key_iv: "5b90e6e3e746c874dbf51621a3d00f34"
server_key_version: "encryptionkey@0"
I20260806 15:58:20.937521 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "759b693e0a89446cbffd7cfd89ab0881"
format_stamp: "Formatted at 2026-08-06 15:58:20 on dist-test-slave-rwrg"
server_key: "18ae30a23b4955957cab872cf73af7af"
server_key_iv: "5b90e6e3e746c874dbf51621a3d00f34"
server_key_version: "encryptionkey@0"
I20260806 15:58:20.941386 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:58:20.943733 13855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:20.945029 13832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:20.945173 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "759b693e0a89446cbffd7cfd89ab0881"
format_stamp: "Formatted at 2026-08-06 15:58:20 on dist-test-slave-rwrg"
server_key: "18ae30a23b4955957cab872cf73af7af"
server_key_iv: "5b90e6e3e746c874dbf51621a3d00f34"
server_key_version: "encryptionkey@0"
I20260806 15:58:20.945272 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:20.957669 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:20.960680 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:20.960873 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:20.967862 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38247
I20260806 15:58:20.967896 13937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38247 every 8 connection(s)
I20260806 15:58:20.968889 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:20.971642 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:20.977139 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881: Bootstrap starting.
I20260806 15:58:20.978718 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 13832
I20260806 15:58:20.978860 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:20.979182 14204 external_mini_cluster.cc:1428] Setting key 32841a8811637fbf5681ad06dd10dd85
I20260806 15:58:20.979884 13938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:20.980845 13938 log.cc:824] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:20.982795 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881: No bootstrap required, opened a new log
I20260806 15:58:20.986187 13938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759b693e0a89446cbffd7cfd89ab0881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38247 } }
I20260806 15:58:20.986421 13938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:20.986502 13938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759b693e0a89446cbffd7cfd89ab0881, State: Initialized, Role: FOLLOWER
I20260806 15:58:20.987001 13938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759b693e0a89446cbffd7cfd89ab0881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38247 } }
I20260806 15:58:20.987175 13938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:20.987241 13938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:20.987336 13938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 15:58:20 dist-test-slave-rwrg krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 15:58:20.988526 13938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759b693e0a89446cbffd7cfd89ab0881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38247 } }
I20260806 15:58:20.988847 13938 leader_election.cc:302] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759b693e0a89446cbffd7cfd89ab0881; no voters:
I20260806 15:58:20.989171 13938 leader_election.cc:288] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:20.989246 13943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:20.989481 13943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [term 1 LEADER]: Becoming Leader. State: Replica: 759b693e0a89446cbffd7cfd89ab0881, State: Running, Role: LEADER
I20260806 15:58:20.989849 13943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759b693e0a89446cbffd7cfd89ab0881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38247 } }
I20260806 15:58:20.990487 13938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:20.991493 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759b693e0a89446cbffd7cfd89ab0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759b693e0a89446cbffd7cfd89ab0881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38247 } } }
I20260806 15:58:20.991655 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:20.992053 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759b693e0a89446cbffd7cfd89ab0881. Latest consensus state: current_term: 1 leader_uuid: "759b693e0a89446cbffd7cfd89ab0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759b693e0a89446cbffd7cfd89ab0881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38247 } } }
I20260806 15:58:20.992238 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:20.995265 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:20.980359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38393 (local address 127.0.0.1:38247)
0806 15:58:20.980851 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:20.980872 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:20.980927 (+ 55us) server_negotiation.cc:407] Connection header received
0806 15:58:20.981836 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:20.981857 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:20.982208 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:20.982544 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:20.983415 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:20.984404 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:20.985323 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:20.985788 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:20.985973 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:20.989282 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:20.989291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:20.989318 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:20.989375 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:20.991715 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:20.992427 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:20.992433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:20.992440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:20.992634 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:20.993067 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:20.993069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:20.993071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:20.993480 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:20.993758 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:20.994028 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 15:58:20.994290 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":185,"threads_started":1}
W20260806 15:58:20.995671 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:20.995927 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:20.996099 13954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:20.997087 13954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:20.999183 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:21.003252 13954 catalog_manager.cc:1347] Generated new cluster ID: c57ec864e4cf42958abf58c16570d5b4
I20260806 15:58:21.003316 13954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:21.010342 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:21.001559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47941 (local address 127.0.0.1:38247)
0806 15:58:21.001700 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:21.001705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:21.001957 (+ 252us) server_negotiation.cc:407] Connection header received
0806 15:58:21.002102 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:21.002105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:21.002163 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:21.002267 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:21.003104 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:21.003747 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:21.004643 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:21.004897 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:21.005027 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:21.006508 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:21.006513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:21.006516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:21.006549 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:21.008143 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:21.008765 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:21.008774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:21.008776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:21.008859 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:21.009225 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:21.009229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:21.009231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:21.009427 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:21.009689 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:21.009984 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 15:58:21.010100 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 15:58:21.013867 13954 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:21.014868 13954 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:21.019470 13954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759b693e0a89446cbffd7cfd89ab0881: Generated new TSK 0
I20260806 15:58:21.020191 13954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:21.023983 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 13832
W20260806 15:58:21.035852 13814 connection.cc:535] client connection to 127.0.0.1:38247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38247): Connection reset by peer (error 104)
2026-08-06T15:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1636 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:21 dist-test-slave-rwrg krb5kdc[13991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:21 dist-test-slave-rwrg krb5kdc[13991](info): set up 2 sockets
Aug 06 15:58:21 dist-test-slave-rwrg krb5kdc[13991](info): commencing operation
krb5kdc: starting...
W20260806 15:58:22.334862 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:22 dist-test-slave-rwrg krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:22.465554 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:22.581933 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:22.582213 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:22.582302 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:22.586752 14019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:22.586812 14019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:22.586831 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:22.586871 14019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:22.586902 14019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:22.592378 14019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:22.593472 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:22.594583 14019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:22.603363 14040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:22.603405 14030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:22.603432 14032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:22.603729 14019 server_base.cc:1034] running on GCE node
I20260806 15:58:22.604228 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:22.605398 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:22.606611 14019 hybrid_clock.cc:648] HybridClock initialized: now 1786031902606591 us; error 31 us; skew 500 ppm
Aug 06 15:58:22 dist-test-slave-rwrg krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:22.609565 14019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:22.610385 14019 webserver.cc:460] Webserver started at http://127.0.0.1:46117/ using document root <none> and password file <none>
I20260806 15:58:22.610898 14019 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:22.610956 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:22.611138 14019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:22.613199 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "adaf150e252f4fc58e8b323d3d5d579d"
format_stamp: "Formatted at 2026-08-06 15:58:22 on dist-test-slave-rwrg"
server_key: "80fcf0069282ca10f73a7e3cf102f4b6"
server_key_iv: "0773686d4e7ef91fe01eeb6a746b3a38"
server_key_version: "encryptionkey@0"
I20260806 15:58:22.613765 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "adaf150e252f4fc58e8b323d3d5d579d"
format_stamp: "Formatted at 2026-08-06 15:58:22 on dist-test-slave-rwrg"
server_key: "80fcf0069282ca10f73a7e3cf102f4b6"
server_key_iv: "0773686d4e7ef91fe01eeb6a746b3a38"
server_key_version: "encryptionkey@0"
I20260806 15:58:22.617463 14019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:58:22.619848 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:22.621170 14019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:22.621328 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "adaf150e252f4fc58e8b323d3d5d579d"
format_stamp: "Formatted at 2026-08-06 15:58:22 on dist-test-slave-rwrg"
server_key: "80fcf0069282ca10f73a7e3cf102f4b6"
server_key_iv: "0773686d4e7ef91fe01eeb6a746b3a38"
server_key_version: "encryptionkey@0"
I20260806 15:58:22.621447 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:22.641235 14019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:22.644779 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:22.645006 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:22.653147 14019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44283
I20260806 15:58:22.653167 14133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44283 every 8 connection(s)
I20260806 15:58:22.654162 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:22.656924 14134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:22.662639 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14019
I20260806 15:58:22.662803 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:22.663173 14134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d: Bootstrap starting.
I20260806 15:58:22.663194 14204 external_mini_cluster.cc:1428] Setting key aad6da2cb8a8e03add105416db28de9c
I20260806 15:58:22.665972 14134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:22.666857 14134 log.cc:824] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:22.669061 14134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d: No bootstrap required, opened a new log
I20260806 15:58:22.673655 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:22.664604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57313 (local address 127.0.0.1:44283)
0806 15:58:22.665092 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:22.665122 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:22.665166 (+ 44us) server_negotiation.cc:407] Connection header received
0806 15:58:22.666023 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:22.666048 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:22.666496 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:22.666860 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:22.667312 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:22.668711 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:22.669733 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:22.670415 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:22.670579 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:22.671476 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:22.671484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:22.671505 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:22.672092 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:22.672177 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:22.672210 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:22.672347 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:22.672552 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 15:58:22.672693 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260806 15:58:22.673473 14134 raft_consensus.cc:348] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaf150e252f4fc58e8b323d3d5d579d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 15:58:22.673759 14134 raft_consensus.cc:374] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:22.673816 14134 raft_consensus.cc:729] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf150e252f4fc58e8b323d3d5d579d, State: Initialized, Role: FOLLOWER
I20260806 15:58:22.674264 14134 consensus_queue.cc:260] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaf150e252f4fc58e8b323d3d5d579d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 15:58:22.674431 14134 raft_consensus.cc:388] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:22.674533 14134 raft_consensus.cc:482] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:22.674656 14134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:22.675822 14134 raft_consensus.cc:504] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaf150e252f4fc58e8b323d3d5d579d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 15:58:22.676285 14134 leader_election.cc:302] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adaf150e252f4fc58e8b323d3d5d579d; no voters:
I20260806 15:58:22.676764 14134 leader_election.cc:288] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:22.676843 14143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:22.677115 14143 raft_consensus.cc:686] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [term 1 LEADER]: Becoming Leader. State: Replica: adaf150e252f4fc58e8b323d3d5d579d, State: Running, Role: LEADER
I20260806 15:58:22.677453 14143 consensus_queue.cc:237] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaf150e252f4fc58e8b323d3d5d579d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 15:58:22.677637 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:22.678155 14134 sys_catalog.cc:564] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:22.678743 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adaf150e252f4fc58e8b323d3d5d579d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaf150e252f4fc58e8b323d3d5d579d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260806 15:58:22.678849 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:22.678946 14144 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [sys.catalog]: SysCatalogTable state changed. Reason: New leader adaf150e252f4fc58e8b323d3d5d579d. Latest consensus state: current_term: 1 leader_uuid: "adaf150e252f4fc58e8b323d3d5d579d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaf150e252f4fc58e8b323d3d5d579d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260806 15:58:22.679075 14144 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:22.679419 14152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:22.683375 14152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:22.691047 14152 catalog_manager.cc:1347] Generated new cluster ID: 92a47490ddc84e96a79efaff4a65eec5
I20260806 15:58:22.691171 14152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:22.696986 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:22.680301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33813 (local address 127.0.0.1:44283)
0806 15:58:22.680441 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:22.680449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:22.681990 (+ 1541us) server_negotiation.cc:407] Connection header received
0806 15:58:22.683474 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:22.683478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:22.683527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:22.683579 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:22.684020 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:22.685848 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:22.689182 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:22.689562 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:22.689653 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:22.691003 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:22.691008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:22.691014 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:22.691032 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:22.691063 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:22.691080 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:22.691189 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:22.696793 (+ 5604us) server_negotiation.cc:299] Negotiation successful
0806 15:58:22.696833 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 15:58:22.713249 14152 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:22.714396 14152 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:22.722643 14152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adaf150e252f4fc58e8b323d3d5d579d: Generated new TSK 0
I20260806 15:58:22.723309 14152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:22.732254 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14019
W20260806 15:58:22.743563 13987 connection.cc:535] client connection to 127.0.0.1:44283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44283): Connection reset by peer (error 104)
2026-08-06T15:58:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1707 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:22 dist-test-slave-rwrg krb5kdc[14184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:22 dist-test-slave-rwrg krb5kdc[14184](info): set up 2 sockets
Aug 06 15:58:22 dist-test-slave-rwrg krb5kdc[14184](info): commencing operation
krb5kdc: starting...
W20260806 15:58:24.052560 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:24 dist-test-slave-rwrg krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:24.183823 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:24.296279 14201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:24.296586 14201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:24.296636 14201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:24.300693 14201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:24.300755 14201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:24.300776 14201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:24.300793 14201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:24.300807 14201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:24.306063 14201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:24.307090 14201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:24.308084 14201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:24.316752 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:24.316764 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:24.316735 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:24.317097 14201 server_base.cc:1034] running on GCE node
I20260806 15:58:24.317682 14201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:24.318785 14201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:24.319985 14201 hybrid_clock.cc:648] HybridClock initialized: now 1786031904319959 us; error 50 us; skew 500 ppm
Aug 06 15:58:24 dist-test-slave-rwrg krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:24.327739 14201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:24.328800 14201 webserver.cc:460] Webserver started at http://127.0.0.1:44737/ using document root <none> and password file <none>
I20260806 15:58:24.329429 14201 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:24.329526 14201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:24.329772 14201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:24.331938 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "36e01ff159d14eb79a6d5b642810bbb1"
format_stamp: "Formatted at 2026-08-06 15:58:24 on dist-test-slave-rwrg"
server_key: "e65cc949e07c7786616948ed28a33a70"
server_key_iv: "62c13cf684fa288e2a9af9b37962acc7"
server_key_version: "encryptionkey@0"
I20260806 15:58:24.332603 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "36e01ff159d14eb79a6d5b642810bbb1"
format_stamp: "Formatted at 2026-08-06 15:58:24 on dist-test-slave-rwrg"
server_key: "e65cc949e07c7786616948ed28a33a70"
server_key_iv: "62c13cf684fa288e2a9af9b37962acc7"
server_key_version: "encryptionkey@0"
I20260806 15:58:24.336329 14201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 15:58:24.338781 14228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:24.339951 14201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:24.340135 14201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "36e01ff159d14eb79a6d5b642810bbb1"
format_stamp: "Formatted at 2026-08-06 15:58:24 on dist-test-slave-rwrg"
server_key: "e65cc949e07c7786616948ed28a33a70"
server_key_iv: "62c13cf684fa288e2a9af9b37962acc7"
server_key_version: "encryptionkey@0"
I20260806 15:58:24.340257 14201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:24.344907 14201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:24.348202 14201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:24.348397 14201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:24.355576 14201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44855
I20260806 15:58:24.355605 14317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44855 every 8 connection(s)
I20260806 15:58:24.356523 14201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:24.359184 14318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:24.360675 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14201
I20260806 15:58:24.360822 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:24.361212 14204 external_mini_cluster.cc:1428] Setting key cc76e363ca565dac4b4362c70289105a
I20260806 15:58:24.365446 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1: Bootstrap starting.
I20260806 15:58:24.366443 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:24.362666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34539 (local address 127.0.0.1:44855)
0806 15:58:24.363095 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:24.363110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:24.363144 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:58:24.363777 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:24.363794 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:24.364060 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:24.364388 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:24.365207 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:24.365213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:24.365233 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:24.365243 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:24.365301 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:24.365321 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:24.365472 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:24.365564 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 15:58:24.365725 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
I20260806 15:58:24.367947 14318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:24.368938 14318 log.cc:824] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:24.369427 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:24.371007 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1: No bootstrap required, opened a new log
I20260806 15:58:24.373292 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:24.371532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53895 (local address 127.0.0.1:44855)
0806 15:58:24.371663 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:24.371667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:24.371876 (+ 209us) server_negotiation.cc:407] Connection header received
0806 15:58:24.371996 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:24.371998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:24.372027 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:24.372093 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:24.372909 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:24.372912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:24.372914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:24.372924 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:24.372948 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:24.372959 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:24.373028 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:24.373141 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 15:58:24.373171 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:58:24.375351 14318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e01ff159d14eb79a6d5b642810bbb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260806 15:58:24.375603 14318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:24.375663 14318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e01ff159d14eb79a6d5b642810bbb1, State: Initialized, Role: FOLLOWER
I20260806 15:58:24.376065 14318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e01ff159d14eb79a6d5b642810bbb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260806 15:58:24.376219 14318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:24.376289 14318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:24.376390 14318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:24.377447 14318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e01ff159d14eb79a6d5b642810bbb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260806 15:58:24.377797 14318 leader_election.cc:302] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36e01ff159d14eb79a6d5b642810bbb1; no voters:
I20260806 15:58:24.378163 14318 leader_election.cc:288] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:24.378250 14332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:24.378437 14332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [term 1 LEADER]: Becoming Leader. State: Replica: 36e01ff159d14eb79a6d5b642810bbb1, State: Running, Role: LEADER
I20260806 15:58:24.378733 14332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e01ff159d14eb79a6d5b642810bbb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260806 15:58:24.379488 14318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:24.380350 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36e01ff159d14eb79a6d5b642810bbb1. Latest consensus state: current_term: 1 leader_uuid: "36e01ff159d14eb79a6d5b642810bbb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e01ff159d14eb79a6d5b642810bbb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260806 15:58:24.380479 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:24.380934 14344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:24.380892 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36e01ff159d14eb79a6d5b642810bbb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e01ff159d14eb79a6d5b642810bbb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260806 15:58:24.381067 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:24.384145 14344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:24.389541 14344 catalog_manager.cc:1347] Generated new cluster ID: 8ba8a50b7d524f988abb0d77140eecf9
I20260806 15:58:24.389613 14344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:24.400169 14344 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:24.400535 14344 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:24.414726 14344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36e01ff159d14eb79a6d5b642810bbb1: Generated new TSK 0
I20260806 15:58:24.415454 14344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:24.425477 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14201
2026-08-06T15:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1693 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:24 dist-test-slave-rwrg krb5kdc[14368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:24 dist-test-slave-rwrg krb5kdc[14368](info): set up 2 sockets
Aug 06 15:58:24 dist-test-slave-rwrg krb5kdc[14368](info): commencing operation
krb5kdc: starting...
W20260806 15:58:25.672111 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:25 dist-test-slave-rwrg krb5kdc[14368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:25.807919 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:25.922276 14389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:25.922550 14389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:25.922591 14389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:25.926610 14389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:25.926671 14389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:25.926692 14389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:25.926707 14389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:25.926723 14389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:25.932729 14389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:25.933764 14389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:25.934828 14389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:25.943442 14396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:25.943542 14398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:25.943709 14389 server_base.cc:1034] running on GCE node
W20260806 15:58:25.943442 14395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:25.944394 14389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:25.945613 14389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:25.946805 14389 hybrid_clock.cc:648] HybridClock initialized: now 1786031905946785 us; error 30 us; skew 500 ppm
Aug 06 15:58:25 dist-test-slave-rwrg krb5kdc[14368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:25.949880 14389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:25.950840 14389 webserver.cc:460] Webserver started at http://127.0.0.1:35161/ using document root <none> and password file <none>
I20260806 15:58:25.951467 14389 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:25.951570 14389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:25.951826 14389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:25.954090 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "03505a08c5dd48649e9a0064049e1b0a"
format_stamp: "Formatted at 2026-08-06 15:58:25 on dist-test-slave-rwrg"
server_key: "0686e3d9aeec4eb9b886c8caef270dcf"
server_key_iv: "2670fa152d50711a57d826a2b44e20fa"
server_key_version: "encryptionkey@0"
I20260806 15:58:25.954722 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "03505a08c5dd48649e9a0064049e1b0a"
format_stamp: "Formatted at 2026-08-06 15:58:25 on dist-test-slave-rwrg"
server_key: "0686e3d9aeec4eb9b886c8caef270dcf"
server_key_iv: "2670fa152d50711a57d826a2b44e20fa"
server_key_version: "encryptionkey@0"
I20260806 15:58:25.959059 14389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 15:58:25.961807 14408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:25.963150 14389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:25.963315 14389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "03505a08c5dd48649e9a0064049e1b0a"
format_stamp: "Formatted at 2026-08-06 15:58:25 on dist-test-slave-rwrg"
server_key: "0686e3d9aeec4eb9b886c8caef270dcf"
server_key_iv: "2670fa152d50711a57d826a2b44e20fa"
server_key_version: "encryptionkey@0"
I20260806 15:58:25.963438 14389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:25.980484 14389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:25.983736 14389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:25.983906 14389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:25.992149 14389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40497
I20260806 15:58:25.992177 14490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40497 every 8 connection(s)
I20260806 15:58:25.993191 14389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:25.995790 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14389
I20260806 15:58:25.995958 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:25.996269 14491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:25.996286 14204 external_mini_cluster.cc:1428] Setting key 2cacc9f384c6649392ace2e0c50d27e5
I20260806 15:58:26.003139 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a: Bootstrap starting.
I20260806 15:58:26.005932 14491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a: Neither blocks nor log segments found. Creating new log.
Aug 06 15:58:26 dist-test-slave-rwrg krb5kdc[14368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 15:58:26.006930 14491 log.cc:824] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:26.009565 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a: No bootstrap required, opened a new log
I20260806 15:58:26.012458 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:25.997809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56813 (local address 127.0.0.1:40497)
0806 15:58:25.998297 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:25.998313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:25.998350 (+ 37us) server_negotiation.cc:407] Connection header received
0806 15:58:25.999223 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:25.999245 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:25.999625 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:25.999992 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:26.000980 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:26.002083 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:26.003098 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:26.003703 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:26.003903 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:26.006988 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:26.006994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:26.007032 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:26.007077 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:26.009390 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:26.009944 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:26.009949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:26.009955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:26.010082 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:26.010399 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:26.010402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:26.010403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:26.010851 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:26.011006 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:26.011319 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 15:58:26.011627 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":159,"threads_started":1}
I20260806 15:58:26.013892 14491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03505a08c5dd48649e9a0064049e1b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 15:58:26.014170 14491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:26.014221 14491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03505a08c5dd48649e9a0064049e1b0a, State: Initialized, Role: FOLLOWER
I20260806 15:58:26.014674 14491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03505a08c5dd48649e9a0064049e1b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 15:58:26.014837 14491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:26.014910 14491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:26.015017 14491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:26.015740 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:26.016273 14491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03505a08c5dd48649e9a0064049e1b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 15:58:26.016682 14491 leader_election.cc:302] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03505a08c5dd48649e9a0064049e1b0a; no voters:
I20260806 15:58:26.017042 14491 leader_election.cc:288] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:26.017241 14498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:26.017505 14498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [term 1 LEADER]: Becoming Leader. State: Replica: 03505a08c5dd48649e9a0064049e1b0a, State: Running, Role: LEADER
I20260806 15:58:26.017866 14498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03505a08c5dd48649e9a0064049e1b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 15:58:26.018532 14491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:26.019250 14498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03505a08c5dd48649e9a0064049e1b0a. Latest consensus state: current_term: 1 leader_uuid: "03505a08c5dd48649e9a0064049e1b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03505a08c5dd48649e9a0064049e1b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260806 15:58:26.019380 14498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:26.019811 14525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:26.019820 14514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03505a08c5dd48649e9a0064049e1b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03505a08c5dd48649e9a0064049e1b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260806 15:58:26.020058 14514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:26.023340 14525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:26.030408 14525 catalog_manager.cc:1347] Generated new cluster ID: c7279ed9f64f40c08676d154e75d7959
I20260806 15:58:26.030488 14525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:26.030817 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:26.018247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50073 (local address 127.0.0.1:40497)
0806 15:58:26.018362 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:26.018367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:26.020850 (+ 2483us) server_negotiation.cc:407] Connection header received
0806 15:58:26.020970 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:26.020974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:26.021027 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:26.021142 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:26.022165 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:26.023026 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:26.024557 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:26.024931 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:26.025027 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:26.026827 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:26.026832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:26.026835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:26.026862 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:26.028816 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:26.029274 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:26.029278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:26.029280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:26.029381 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:26.029688 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:26.029693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:26.029695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:26.029893 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:26.030041 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:26.030523 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 15:58:26.030659 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 15:58:26.041793 14525 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:26.042769 14525 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:26.051141 14525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03505a08c5dd48649e9a0064049e1b0a: Generated new TSK 0
I20260806 15:58:26.051872 14525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:26.059266 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14389
2026-08-06T15:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1640 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:26 dist-test-slave-rwrg krb5kdc[14543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:26 dist-test-slave-rwrg krb5kdc[14543](info): set up 2 sockets
Aug 06 15:58:26 dist-test-slave-rwrg krb5kdc[14543](info): commencing operation
krb5kdc: starting...
W20260806 15:58:27.359078 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:27 dist-test-slave-rwrg krb5kdc[14543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:27.488029 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:27.600970 14568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:27.601287 14568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:27.601372 14568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:27.605453 14568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:27.605557 14568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:27.605594 14568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:27.605631 14568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:27.605669 14568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:27.610973 14568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:27.612114 14568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:27.613168 14568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:27.621668 14581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:27.621680 14576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:27.621764 14568 server_base.cc:1034] running on GCE node
W20260806 15:58:27.621681 14575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:27.622438 14568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:27.623538 14568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:27.624768 14568 hybrid_clock.cc:648] HybridClock initialized: now 1786031907624753 us; error 25 us; skew 500 ppm
Aug 06 15:58:27 dist-test-slave-rwrg krb5kdc[14543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:27.627774 14568 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:27.628808 14568 webserver.cc:460] Webserver started at http://127.0.0.1:34163/ using document root <none> and password file <none>
I20260806 15:58:27.629388 14568 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:27.629485 14568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:27.629745 14568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:27.631835 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "6a29a711071644138954430f88d9b736"
format_stamp: "Formatted at 2026-08-06 15:58:27 on dist-test-slave-rwrg"
server_key: "38b56dbc2478fd4625943883f97673a3"
server_key_iv: "83f44f0817e05cc9b12df5c7e72610dd"
server_key_version: "encryptionkey@0"
I20260806 15:58:27.632426 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "6a29a711071644138954430f88d9b736"
format_stamp: "Formatted at 2026-08-06 15:58:27 on dist-test-slave-rwrg"
server_key: "38b56dbc2478fd4625943883f97673a3"
server_key_iv: "83f44f0817e05cc9b12df5c7e72610dd"
server_key_version: "encryptionkey@0"
I20260806 15:58:27.636173 14568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:27.638671 14589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:27.639914 14568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:27.640085 14568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "6a29a711071644138954430f88d9b736"
format_stamp: "Formatted at 2026-08-06 15:58:27 on dist-test-slave-rwrg"
server_key: "38b56dbc2478fd4625943883f97673a3"
server_key_iv: "83f44f0817e05cc9b12df5c7e72610dd"
server_key_version: "encryptionkey@0"
I20260806 15:58:27.640208 14568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:27.659989 14568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:27.663479 14568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:27.663703 14568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:27.671403 14568 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43003
I20260806 15:58:27.671429 14663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43003 every 8 connection(s)
I20260806 15:58:27.672385 14568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:27.674975 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14568
I20260806 15:58:27.675199 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:27.675423 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:27.675586 14204 external_mini_cluster.cc:1428] Setting key 129f47960e52d76c0fbe12a9d35c5989
I20260806 15:58:27.681993 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736: Bootstrap starting.
I20260806 15:58:27.684221 14667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:27.685204 14667 log.cc:824] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:27.685648 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:27.677141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42891 (local address 127.0.0.1:43003)
0806 15:58:27.677646 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:27.677665 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:27.677707 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:58:27.678712 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:27.678734 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:27.679105 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:27.679462 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:27.679896 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:27.680979 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:27.681931 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:27.682467 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:27.682651 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:27.683565 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:27.683570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:27.683583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:27.683969 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:27.684035 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:27.684060 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:27.684177 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:27.684433 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 15:58:27.684688 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":159,"threads_started":1}
I20260806 15:58:27.687354 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736: No bootstrap required, opened a new log
I20260806 15:58:27.688961 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:27.692030 14667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a29a711071644138954430f88d9b736" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260806 15:58:27.692314 14667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:27.692385 14667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a29a711071644138954430f88d9b736, State: Initialized, Role: FOLLOWER
I20260806 15:58:27.692921 14667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a29a711071644138954430f88d9b736" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260806 15:58:27.693089 14667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:27.693153 14667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:27.693248 14667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:27.694248 14667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a29a711071644138954430f88d9b736" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260806 15:58:27.694619 14667 leader_election.cc:302] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a29a711071644138954430f88d9b736; no voters:
I20260806 15:58:27.695021 14667 leader_election.cc:288] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:27.695111 14689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:27.695323 14689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [term 1 LEADER]: Becoming Leader. State: Replica: 6a29a711071644138954430f88d9b736, State: Running, Role: LEADER
I20260806 15:58:27.695626 14689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a29a711071644138954430f88d9b736" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260806 15:58:27.696189 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:27.691475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54165 (local address 127.0.0.1:43003)
0806 15:58:27.691650 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:27.691655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:27.691826 (+ 171us) server_negotiation.cc:407] Connection header received
0806 15:58:27.691928 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:27.691931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:27.691980 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:27.692149 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:27.692597 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:27.693346 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:27.694276 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:27.694631 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:27.694748 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:27.695680 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:27.695683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:27.695686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:27.695698 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:27.695726 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:27.695742 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:27.695794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:27.695987 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 15:58:27.696038 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 15:58:27.696403 14667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:27.696938 14689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a29a711071644138954430f88d9b736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a29a711071644138954430f88d9b736" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } } }
I20260806 15:58:27.697129 14689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:27.697388 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a29a711071644138954430f88d9b736. Latest consensus state: current_term: 1 leader_uuid: "6a29a711071644138954430f88d9b736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a29a711071644138954430f88d9b736" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } } }
I20260806 15:58:27.697561 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:27.697602 14701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:27.700953 14701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:27.709266 14701 catalog_manager.cc:1347] Generated new cluster ID: 70fe2ebaebb5488480e757c48a5d5a56
I20260806 15:58:27.709355 14701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:27.719648 14701 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:27.720680 14701 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:27.729617 14701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a29a711071644138954430f88d9b736: Generated new TSK 0
I20260806 15:58:27.730360 14701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:27.743904 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14568
2026-08-06T15:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1677 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:27 dist-test-slave-rwrg krb5kdc[14725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:27 dist-test-slave-rwrg krb5kdc[14725](info): set up 2 sockets
Aug 06 15:58:27 dist-test-slave-rwrg krb5kdc[14725](info): commencing operation
krb5kdc: starting...
W20260806 15:58:28.991097 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:29 dist-test-slave-rwrg krb5kdc[14725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:29.122892 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:29.241231 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:29.241528 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:29.241647 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:29.245568 14741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:29.245651 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:29.245687 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:29.245745 14741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:29.245781 14741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:29.251057 14741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:29.252120 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:29.253198 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:29.261688 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:29.261705 14749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:29.265672 14741 server_base.cc:1034] running on GCE node
W20260806 15:58:29.267244 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:29.267899 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:29.269346 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:29.270565 14741 hybrid_clock.cc:648] HybridClock initialized: now 1786031909270541 us; error 46 us; skew 500 ppm
Aug 06 15:58:29 dist-test-slave-rwrg krb5kdc[14725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:29.273511 14741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:29.274412 14741 webserver.cc:460] Webserver started at http://127.0.0.1:34181/ using document root <none> and password file <none>
I20260806 15:58:29.274963 14741 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:29.275022 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:29.275238 14741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:29.277320 14741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "111b793087e24088a6c251fcadeab677"
format_stamp: "Formatted at 2026-08-06 15:58:29 on dist-test-slave-rwrg"
server_key: "7fe928f57d52d5f21bdcd1d74d168db0"
server_key_iv: "e6f9ddb55b7873f84073dddd82778253"
server_key_version: "encryptionkey@0"
I20260806 15:58:29.277871 14741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "111b793087e24088a6c251fcadeab677"
format_stamp: "Formatted at 2026-08-06 15:58:29 on dist-test-slave-rwrg"
server_key: "7fe928f57d52d5f21bdcd1d74d168db0"
server_key_iv: "e6f9ddb55b7873f84073dddd82778253"
server_key_version: "encryptionkey@0"
I20260806 15:58:29.281749 14741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 15:58:29.284368 14769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:29.285779 14741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:29.285928 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "111b793087e24088a6c251fcadeab677"
format_stamp: "Formatted at 2026-08-06 15:58:29 on dist-test-slave-rwrg"
server_key: "7fe928f57d52d5f21bdcd1d74d168db0"
server_key_iv: "e6f9ddb55b7873f84073dddd82778253"
server_key_version: "encryptionkey@0"
I20260806 15:58:29.286057 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:29.290711 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:29.293920 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:29.294107 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:29.302028 14741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32987
I20260806 15:58:29.302090 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32987 every 8 connection(s)
I20260806 15:58:29.303020 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:29.305835 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:29.309679 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14741
I20260806 15:58:29.309862 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:29.310215 14204 external_mini_cluster.cc:1428] Setting key 55c302df5778ffd831f6fbfd673ca79a
I20260806 15:58:29.311859 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677: Bootstrap starting.
I20260806 15:58:29.314724 14859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:29.315711 14859 log.cc:824] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:29.316471 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:29.311853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40797 (local address 127.0.0.1:32987)
0806 15:58:29.312373 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:29.312392 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:29.312435 (+ 43us) server_negotiation.cc:407] Connection header received
0806 15:58:29.313359 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:29.313383 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:29.313754 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:29.314140 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:29.315066 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:29.315072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:29.315089 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:29.315101 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:29.315164 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:29.315190 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:29.315298 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:29.315450 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 15:58:29.315627 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":167,"threads_started":1}
I20260806 15:58:29.318506 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677: No bootstrap required, opened a new log
I20260806 15:58:29.319308 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:29.323179 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:29.321287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52255 (local address 127.0.0.1:32987)
0806 15:58:29.321407 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:29.321411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:29.321682 (+ 271us) server_negotiation.cc:407] Connection header received
0806 15:58:29.321832 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:29.321835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:29.321874 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:29.322000 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:29.322773 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:29.322776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:29.322778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:29.322790 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:29.322815 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:29.322828 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:29.322919 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:29.322992 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 15:58:29.323030 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 15:58:29.324842 14859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111b793087e24088a6c251fcadeab677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260806 15:58:29.325173 14859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:29.325244 14859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111b793087e24088a6c251fcadeab677, State: Initialized, Role: FOLLOWER
I20260806 15:58:29.325793 14859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111b793087e24088a6c251fcadeab677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260806 15:58:29.325980 14859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:29.326061 14859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:29.326182 14859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:29.327844 14859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111b793087e24088a6c251fcadeab677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260806 15:58:29.328284 14859 leader_election.cc:302] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 111b793087e24088a6c251fcadeab677; no voters:
I20260806 15:58:29.328773 14859 leader_election.cc:288] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:29.328897 14871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:29.329273 14871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [term 1 LEADER]: Becoming Leader. State: Replica: 111b793087e24088a6c251fcadeab677, State: Running, Role: LEADER
I20260806 15:58:29.329697 14871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111b793087e24088a6c251fcadeab677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260806 15:58:29.330718 14859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:29.331421 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 111b793087e24088a6c251fcadeab677. Latest consensus state: current_term: 1 leader_uuid: "111b793087e24088a6c251fcadeab677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111b793087e24088a6c251fcadeab677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } } }
I20260806 15:58:29.331558 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:29.331908 14873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "111b793087e24088a6c251fcadeab677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111b793087e24088a6c251fcadeab677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } } }
I20260806 15:58:29.332113 14873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:29.332580 14878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:29.335804 14878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:29.341748 14878 catalog_manager.cc:1347] Generated new cluster ID: 4b12f8c874594e9fa5cbc55bae86e5b7
I20260806 15:58:29.341820 14878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:29.354312 14878 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:29.354660 14878 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:29.366035 14878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 111b793087e24088a6c251fcadeab677: Generated new TSK 0
I20260806 15:58:29.366735 14878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:29.383214 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14741
2026-08-06T15:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1643 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:29 dist-test-slave-rwrg krb5kdc[14908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:29 dist-test-slave-rwrg krb5kdc[14908](info): set up 2 sockets
Aug 06 15:58:29 dist-test-slave-rwrg krb5kdc[14908](info): commencing operation
krb5kdc: starting...
W20260806 15:58:30.653818 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:30 dist-test-slave-rwrg krb5kdc[14908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:30.783937 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:30.897065 14931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:30.897322 14931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:30.897362 14931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:30.901293 14931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:30.901355 14931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:30.901374 14931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:30.901391 14931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:30.901405 14931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:30.906639 14931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:30.907681 14931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:30.908684 14931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:30.917464 14937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:30.917503 14941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:30.917503 14931 server_base.cc:1034] running on GCE node
W20260806 15:58:30.917680 14938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:30.918385 14931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:30.919433 14931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:30.920605 14931 hybrid_clock.cc:648] HybridClock initialized: now 1786031910920577 us; error 42 us; skew 500 ppm
Aug 06 15:58:30 dist-test-slave-rwrg krb5kdc[14908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:30.923332 14931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:30.924263 14931 webserver.cc:460] Webserver started at http://127.0.0.1:37937/ using document root <none> and password file <none>
I20260806 15:58:30.924861 14931 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:30.924952 14931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:30.925180 14931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:30.927191 14931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "84fa36501bfe48178613c352d2cb125b"
format_stamp: "Formatted at 2026-08-06 15:58:30 on dist-test-slave-rwrg"
server_key: "e45cde2357e67e72642e9cce831c2a42"
server_key_iv: "67c725d3ff3b3e439a29891a0af4f044"
server_key_version: "encryptionkey@0"
I20260806 15:58:30.927779 14931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "84fa36501bfe48178613c352d2cb125b"
format_stamp: "Formatted at 2026-08-06 15:58:30 on dist-test-slave-rwrg"
server_key: "e45cde2357e67e72642e9cce831c2a42"
server_key_iv: "67c725d3ff3b3e439a29891a0af4f044"
server_key_version: "encryptionkey@0"
I20260806 15:58:30.931752 14931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:58:30.934757 14951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:30.936061 14931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:30.936244 14931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "84fa36501bfe48178613c352d2cb125b"
format_stamp: "Formatted at 2026-08-06 15:58:30 on dist-test-slave-rwrg"
server_key: "e45cde2357e67e72642e9cce831c2a42"
server_key_iv: "67c725d3ff3b3e439a29891a0af4f044"
server_key_version: "encryptionkey@0"
I20260806 15:58:30.936362 14931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:30.962364 14931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:30.965253 14931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:30.965467 14931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:30.972795 14931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45909
I20260806 15:58:30.972821 15034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45909 every 8 connection(s)
I20260806 15:58:30.973745 14931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:30.976394 15036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:30.981341 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 14931
I20260806 15:58:30.981495 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:30.981801 14204 external_mini_cluster.cc:1428] Setting key ce76f4097dcc54584e04b6e4a9360068
I20260806 15:58:30.982174 15036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b: Bootstrap starting.
I20260806 15:58:30.984778 15036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:30.985520 15036 log.cc:824] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:30.987376 15036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b: No bootstrap required, opened a new log
Aug 06 15:58:30 dist-test-slave-rwrg krb5kdc[14908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 15:58:30.991087 15036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa36501bfe48178613c352d2cb125b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260806 15:58:30.991329 15036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:30.991394 15036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fa36501bfe48178613c352d2cb125b, State: Initialized, Role: FOLLOWER
I20260806 15:58:30.991942 15036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa36501bfe48178613c352d2cb125b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260806 15:58:30.992120 15036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:30.992189 15036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:30.992290 15036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:30.993230 15036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa36501bfe48178613c352d2cb125b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260806 15:58:30.993609 15036 leader_election.cc:302] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84fa36501bfe48178613c352d2cb125b; no voters:
I20260806 15:58:30.993911 15036 leader_election.cc:288] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:30.994055 15044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:30.994302 15044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [term 1 LEADER]: Becoming Leader. State: Replica: 84fa36501bfe48178613c352d2cb125b, State: Running, Role: LEADER
I20260806 15:58:30.994656 15044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa36501bfe48178613c352d2cb125b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260806 15:58:30.995386 15036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:30.995888 15044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84fa36501bfe48178613c352d2cb125b. Latest consensus state: current_term: 1 leader_uuid: "84fa36501bfe48178613c352d2cb125b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa36501bfe48178613c352d2cb125b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260806 15:58:30.996078 15044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:30.996340 15047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84fa36501bfe48178613c352d2cb125b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa36501bfe48178613c352d2cb125b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260806 15:58:30.996536 15047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:30.996492 15051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:30.996941 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:30.983236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:38551 (local address 127.0.0.1:45909)
0806 15:58:30.983675 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:30.983690 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:30.983724 (+ 34us) server_negotiation.cc:407] Connection header received
0806 15:58:30.984329 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:30.984346 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:30.984773 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:30.985142 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:30.986000 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:30.986951 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:30.987887 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:30.988327 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:30.988490 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:30.991065 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:30.991070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:30.991083 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:30.991114 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:30.993088 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:30.993695 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:30.993701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:30.993706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:30.993847 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:30.994278 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:30.994283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:30.994286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:30.994911 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:30.995133 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:30.995449 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 15:58:30.995895 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260806 15:58:30.999830 15051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:31.000116 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:31.006440 15051 catalog_manager.cc:1347] Generated new cluster ID: ead3fee216ff447eaa73bb173911a9d4
I20260806 15:58:31.006510 15051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:31.012872 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:31.002275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:38669 (local address 127.0.0.1:45909)
0806 15:58:31.002387 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:31.002392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:31.002650 (+ 258us) server_negotiation.cc:407] Connection header received
0806 15:58:31.002764 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:31.002768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:31.002808 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:31.002901 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:31.003521 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:31.004102 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:31.005307 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:31.005645 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:31.005751 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:31.009396 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:31.009400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:31.009403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:31.009427 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:31.010984 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:31.011479 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:31.011483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:31.011484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:31.011569 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:31.011904 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:31.011907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:31.011908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:31.012095 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:31.012257 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:31.012571 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 15:58:31.012671 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 15:58:31.030774 15051 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:31.031767 15051 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:31.037246 15051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84fa36501bfe48178613c352d2cb125b: Generated new TSK 0
I20260806 15:58:31.037992 15051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:31.048136 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 14931
W20260806 15:58:31.064210 14906 connection.cc:535] client connection to 127.0.0.1:45909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45909): Connection reset by peer (error 104)
2026-08-06T15:58:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1666 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:31 dist-test-slave-rwrg krb5kdc[15084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:31 dist-test-slave-rwrg krb5kdc[15084](info): set up 2 sockets
Aug 06 15:58:31 dist-test-slave-rwrg krb5kdc[15084](info): commencing operation
krb5kdc: starting...
W20260806 15:58:32.316237 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:32 dist-test-slave-rwrg krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:32.445941 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:32.557188 15101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:32.557483 15101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:32.557557 15101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:32.561707 15101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:32.561815 15101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:32.561853 15101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:32.561913 15101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:32.561951 15101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:32.567063 15101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:32.568699 15101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:32.569968 15101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:58:32.578639 15101 server_base.cc:1034] running on GCE node
W20260806 15:58:32.578586 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:32.578610 15118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:32.578612 15121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:32.579444 15101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:32.580582 15101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:32.581794 15101 hybrid_clock.cc:648] HybridClock initialized: now 1786031912581775 us; error 36 us; skew 500 ppm
Aug 06 15:58:32 dist-test-slave-rwrg krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:32.584555 15101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:32.585491 15101 webserver.cc:460] Webserver started at http://127.0.0.1:42815/ using document root <none> and password file <none>
I20260806 15:58:32.586086 15101 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:32.586181 15101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:32.586411 15101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:32.588486 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "1f4cede0a65c467e96ec18a1063a9732"
format_stamp: "Formatted at 2026-08-06 15:58:32 on dist-test-slave-rwrg"
server_key: "5e7e026255b4b11dbc3e052c93035338"
server_key_iv: "c3ff6b1f50ba72192c44919c923d826b"
server_key_version: "encryptionkey@0"
I20260806 15:58:32.589144 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "1f4cede0a65c467e96ec18a1063a9732"
format_stamp: "Formatted at 2026-08-06 15:58:32 on dist-test-slave-rwrg"
server_key: "5e7e026255b4b11dbc3e052c93035338"
server_key_iv: "c3ff6b1f50ba72192c44919c923d826b"
server_key_version: "encryptionkey@0"
I20260806 15:58:32.593139 15101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 15:58:32.595777 15133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:32.597071 15101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:32.597227 15101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "1f4cede0a65c467e96ec18a1063a9732"
format_stamp: "Formatted at 2026-08-06 15:58:32 on dist-test-slave-rwrg"
server_key: "5e7e026255b4b11dbc3e052c93035338"
server_key_iv: "c3ff6b1f50ba72192c44919c923d826b"
server_key_version: "encryptionkey@0"
I20260806 15:58:32.597345 15101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:32.612387 15101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:32.615474 15101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:32.615708 15101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:32.623510 15101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45417
I20260806 15:58:32.623524 15217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45417 every 8 connection(s)
I20260806 15:58:32.624549 15101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:32.627489 15219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:32.632727 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15101
I20260806 15:58:32.632889 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:32.633210 14204 external_mini_cluster.cc:1428] Setting key 745428487f9e9b3796142f06b9297912
I20260806 15:58:32.633442 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732: Bootstrap starting.
I20260806 15:58:32.636087 15219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:32.637053 15219 log.cc:824] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:32.639312 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732: No bootstrap required, opened a new log
I20260806 15:58:32.642437 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:32.634567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:57385 (local address 127.0.0.1:45417)
0806 15:58:32.635009 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:32.635026 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:32.635062 (+ 36us) server_negotiation.cc:407] Connection header received
0806 15:58:32.635699 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:32.635715 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:32.636057 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:32.636394 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:32.636809 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:32.637830 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:32.639000 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:32.639508 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:32.639630 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:32.640660 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:32.640665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:32.640678 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:32.641065 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:32.641122 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:32.641144 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:32.641245 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:32.641469 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 15:58:32.641669 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":153,"threads_started":1}
I20260806 15:58:32.643440 15219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4cede0a65c467e96ec18a1063a9732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260806 15:58:32.643671 15219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:32.643761 15219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4cede0a65c467e96ec18a1063a9732, State: Initialized, Role: FOLLOWER
I20260806 15:58:32.644205 15219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4cede0a65c467e96ec18a1063a9732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260806 15:58:32.644362 15219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:32.644452 15219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:32.644591 15219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:32.645680 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:32.645814 15219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4cede0a65c467e96ec18a1063a9732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260806 15:58:32.646210 15219 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4cede0a65c467e96ec18a1063a9732; no voters:
I20260806 15:58:32.646617 15219 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:32.646701 15227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:32.646994 15227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4cede0a65c467e96ec18a1063a9732, State: Running, Role: LEADER
I20260806 15:58:32.647397 15227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4cede0a65c467e96ec18a1063a9732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260806 15:58:32.648094 15219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:32.648685 15227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4cede0a65c467e96ec18a1063a9732. Latest consensus state: current_term: 1 leader_uuid: "1f4cede0a65c467e96ec18a1063a9732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4cede0a65c467e96ec18a1063a9732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260806 15:58:32.648865 15227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:32.649132 15228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4cede0a65c467e96ec18a1063a9732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4cede0a65c467e96ec18a1063a9732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260806 15:58:32.649312 15228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:32.649338 15246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:32.653275 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:32.648009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:39027 (local address 127.0.0.1:45417)
0806 15:58:32.648126 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:32.648130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:32.648577 (+ 447us) server_negotiation.cc:407] Connection header received
0806 15:58:32.648778 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:32.648784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:32.648838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:32.648891 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:32.649356 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:32.650118 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:32.651470 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:32.651702 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:32.651836 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:32.652742 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:32.652745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:32.652748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:32.652761 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:32.652794 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:32.652804 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:32.652903 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:32.653074 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 15:58:32.653117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 15:58:32.653896 15246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:32.659574 15246 catalog_manager.cc:1347] Generated new cluster ID: 81621ad2f4994569a912d6f37bb3bb81
I20260806 15:58:32.659658 15246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:32.668421 15246 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:32.669394 15246 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:32.680063 15246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4cede0a65c467e96ec18a1063a9732: Generated new TSK 0
I20260806 15:58:32.681077 15246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:32.690310 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15101
W20260806 15:58:32.709800 15082 connection.cc:535] client connection to 127.0.0.1:45417 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45417): Connection reset by peer (error 104)
2026-08-06T15:58:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1645 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:32 dist-test-slave-rwrg krb5kdc[15272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:32 dist-test-slave-rwrg krb5kdc[15272](info): set up 2 sockets
Aug 06 15:58:32 dist-test-slave-rwrg krb5kdc[15272](info): commencing operation
krb5kdc: starting...
W20260806 15:58:33.999887 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:34 dist-test-slave-rwrg krb5kdc[15272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:34.131695 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:34.242899 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:34.243206 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:34.243286 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:34.247202 15295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:34.247290 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:34.247328 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:34.247382 15295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:34.247421 15295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:34.252720 15295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:34.253764 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:34.254772 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:34.263119 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:34.263162 15304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:34.263188 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:34.263417 15295 server_base.cc:1034] running on GCE node
I20260806 15:58:34.263964 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:34.265110 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:34.266305 15295 hybrid_clock.cc:648] HybridClock initialized: now 1786031914266278 us; error 40 us; skew 500 ppm
Aug 06 15:58:34 dist-test-slave-rwrg krb5kdc[15272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:34.269089 15295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:34.269995 15295 webserver.cc:460] Webserver started at http://127.0.0.1:36473/ using document root <none> and password file <none>
I20260806 15:58:34.270548 15295 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:34.270635 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:34.270874 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:34.272931 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "48812a8555a44651a83fbb93248dc522"
format_stamp: "Formatted at 2026-08-06 15:58:34 on dist-test-slave-rwrg"
server_key: "0e7f3ec8b1cc9b63203fb3d494c6deae"
server_key_iv: "2bf26bb3e9be9109bff8012c89c6741c"
server_key_version: "encryptionkey@0"
I20260806 15:58:34.273495 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "48812a8555a44651a83fbb93248dc522"
format_stamp: "Formatted at 2026-08-06 15:58:34 on dist-test-slave-rwrg"
server_key: "0e7f3ec8b1cc9b63203fb3d494c6deae"
server_key_iv: "2bf26bb3e9be9109bff8012c89c6741c"
server_key_version: "encryptionkey@0"
I20260806 15:58:34.277210 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:34.279569 15322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:34.280881 15295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:34.281049 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "48812a8555a44651a83fbb93248dc522"
format_stamp: "Formatted at 2026-08-06 15:58:34 on dist-test-slave-rwrg"
server_key: "0e7f3ec8b1cc9b63203fb3d494c6deae"
server_key_iv: "2bf26bb3e9be9109bff8012c89c6741c"
server_key_version: "encryptionkey@0"
I20260806 15:58:34.281173 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:34.286026 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:34.289049 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:34.289429 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:34.296752 15295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35061
I20260806 15:58:34.296780 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35061 every 8 connection(s)
I20260806 15:58:34.297721 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:34.298308 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15295
I20260806 15:58:34.298470 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:34.298763 14204 external_mini_cluster.cc:1428] Setting key 245514e29be6b1490a1599febeecf484
I20260806 15:58:34.300992 15420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:34.304028 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:34.300239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:37481 (local address 127.0.0.1:35061)
0806 15:58:34.300717 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:34.300743 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:34.300783 (+ 40us) server_negotiation.cc:407] Connection header received
0806 15:58:34.301550 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:34.301567 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:34.301881 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:34.302161 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:34.302935 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:34.302939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:34.302950 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:34.302959 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:34.303008 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:34.303025 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:34.303135 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:34.303243 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 15:58:34.303359 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
I20260806 15:58:34.306957 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:34.307581 15420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522: Bootstrap starting.
I20260806 15:58:34.310231 15420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:34.310861 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:34.309143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:52041 (local address 127.0.0.1:35061)
0806 15:58:34.309288 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:34.309292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:34.309468 (+ 176us) server_negotiation.cc:407] Connection header received
0806 15:58:34.309587 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:34.309612 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:34.309641 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:34.309707 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:34.310434 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:34.310437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:34.310439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:34.310449 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:34.310473 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:34.310482 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:34.310553 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:34.310684 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 15:58:34.310718 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 15:58:34.311190 15420 log.cc:824] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:34.313349 15420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522: No bootstrap required, opened a new log
I20260806 15:58:34.316958 15420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48812a8555a44651a83fbb93248dc522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260806 15:58:34.317173 15420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:34.317207 15420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48812a8555a44651a83fbb93248dc522, State: Initialized, Role: FOLLOWER
I20260806 15:58:34.317618 15420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48812a8555a44651a83fbb93248dc522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260806 15:58:34.317741 15420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:34.317787 15420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:34.317862 15420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:34.318709 15420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48812a8555a44651a83fbb93248dc522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260806 15:58:34.319000 15420 leader_election.cc:302] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48812a8555a44651a83fbb93248dc522; no voters:
I20260806 15:58:34.319242 15420 leader_election.cc:288] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:34.319386 15436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:34.319659 15436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [term 1 LEADER]: Becoming Leader. State: Replica: 48812a8555a44651a83fbb93248dc522, State: Running, Role: LEADER
I20260806 15:58:34.320000 15436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48812a8555a44651a83fbb93248dc522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260806 15:58:34.320624 15420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:34.322273 15438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48812a8555a44651a83fbb93248dc522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48812a8555a44651a83fbb93248dc522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } } }
I20260806 15:58:34.322453 15438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:34.322487 15439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48812a8555a44651a83fbb93248dc522. Latest consensus state: current_term: 1 leader_uuid: "48812a8555a44651a83fbb93248dc522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48812a8555a44651a83fbb93248dc522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } } }
I20260806 15:58:34.322649 15439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:34.323289 15451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:34.325888 15451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:34.330912 15451 catalog_manager.cc:1347] Generated new cluster ID: a8574ccf387441cb84fee10bb2fa0615
I20260806 15:58:34.331041 15451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:34.351830 15451 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:34.352224 15451 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:34.362516 15451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48812a8555a44651a83fbb93248dc522: Generated new TSK 0
I20260806 15:58:34.363242 15451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:34.372886 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15295
2026-08-06T15:58:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1674 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:34 dist-test-slave-rwrg krb5kdc[15473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:34 dist-test-slave-rwrg krb5kdc[15473](info): set up 2 sockets
Aug 06 15:58:34 dist-test-slave-rwrg krb5kdc[15473](info): commencing operation
krb5kdc: starting...
W20260806 15:58:35.613437 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:35 dist-test-slave-rwrg krb5kdc[15473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:35.743201 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:35.856698 15492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:35.856961 15492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:35.857035 15492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:35.860877 15492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:35.860929 15492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:35.860946 15492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:35.860992 15492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:35.861024 15492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:35.866230 15492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:35.867316 15492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:35.868331 15492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:35.876797 15503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:35.876855 15506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:35.876946 15502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:35.877157 15492 server_base.cc:1034] running on GCE node
I20260806 15:58:35.877737 15492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:35.878825 15492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:35.880016 15492 hybrid_clock.cc:648] HybridClock initialized: now 1786031915880004 us; error 23 us; skew 500 ppm
Aug 06 15:58:35 dist-test-slave-rwrg krb5kdc[15473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:35.882788 15492 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:35.883628 15492 webserver.cc:460] Webserver started at http://127.0.0.1:37113/ using document root <none> and password file <none>
I20260806 15:58:35.884177 15492 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:35.884265 15492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:35.884527 15492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:35.886490 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "2f912e8da3154fe0b92cc6aa30325ddc"
format_stamp: "Formatted at 2026-08-06 15:58:35 on dist-test-slave-rwrg"
server_key: "176bfd4eb069c9a90bd9dca94ed541a6"
server_key_iv: "9136cea898c31870daf2fdf6de41ece3"
server_key_version: "encryptionkey@0"
I20260806 15:58:35.887027 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "2f912e8da3154fe0b92cc6aa30325ddc"
format_stamp: "Formatted at 2026-08-06 15:58:35 on dist-test-slave-rwrg"
server_key: "176bfd4eb069c9a90bd9dca94ed541a6"
server_key_iv: "9136cea898c31870daf2fdf6de41ece3"
server_key_version: "encryptionkey@0"
I20260806 15:58:35.890816 15492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:35.893399 15517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:35.894734 15492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:35.894877 15492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "2f912e8da3154fe0b92cc6aa30325ddc"
format_stamp: "Formatted at 2026-08-06 15:58:35 on dist-test-slave-rwrg"
server_key: "176bfd4eb069c9a90bd9dca94ed541a6"
server_key_iv: "9136cea898c31870daf2fdf6de41ece3"
server_key_version: "encryptionkey@0"
I20260806 15:58:35.894994 15492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:35.908612 15492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:35.911724 15492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:35.911962 15492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:35.919480 15492 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38277
I20260806 15:58:35.919507 15602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38277 every 8 connection(s)
I20260806 15:58:35.920555 15492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:35.923337 15603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:35.929334 15603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc: Bootstrap starting.
I20260806 15:58:35.930225 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15492
I20260806 15:58:35.930383 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:35.930756 14204 external_mini_cluster.cc:1428] Setting key 3d41d7649a43e38321f3f68364ff6b8c
I20260806 15:58:35.932544 15603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:35.933333 15603 log.cc:824] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:35.935333 15603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc: No bootstrap required, opened a new log
I20260806 15:58:35.938788 15603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38277 } }
I20260806 15:58:35.939046 15603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:35.939090 15603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f912e8da3154fe0b92cc6aa30325ddc, State: Initialized, Role: FOLLOWER
I20260806 15:58:35.939630 15603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38277 } }
I20260806 15:58:35.939801 15603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:35.939900 15603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:35.940042 15603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 0 FOLLOWER]: Advancing to term 1
Aug 06 15:58:35 dist-test-slave-rwrg krb5kdc[15473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 15:58:35.941219 15603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38277 } }
I20260806 15:58:35.941583 15603 leader_election.cc:302] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f912e8da3154fe0b92cc6aa30325ddc; no voters:
I20260806 15:58:35.941951 15603 leader_election.cc:288] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:35.942037 15610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:35.942301 15610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [term 1 LEADER]: Becoming Leader. State: Replica: 2f912e8da3154fe0b92cc6aa30325ddc, State: Running, Role: LEADER
I20260806 15:58:35.942672 15610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38277 } }
I20260806 15:58:35.943395 15603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:35.944669 15613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f912e8da3154fe0b92cc6aa30325ddc. Latest consensus state: current_term: 1 leader_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38277 } } }
I20260806 15:58:35.944876 15613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:35.944464 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f912e8da3154fe0b92cc6aa30325ddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38277 } } }
I20260806 15:58:35.945344 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:35.946066 15628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:35.947433 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:35.932022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:60655 (local address 127.0.0.1:38277)
0806 15:58:35.932592 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:35.932630 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:35.932688 (+ 58us) server_negotiation.cc:407] Connection header received
0806 15:58:35.933475 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:35.933506 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:35.933864 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:35.934214 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:35.935081 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:35.936123 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:35.937091 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:35.937600 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:35.937788 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:35.941041 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:35.941053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:35.941068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:35.941097 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:35.943546 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:35.944244 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:35.944251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:35.944259 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:35.944410 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:35.944955 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:35.944959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:35.944961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:35.945498 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:35.945729 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:35.946466 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 15:58:35.946675 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":160,"threads_started":1}
I20260806 15:58:35.949532 15628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:35.951050 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:35.958551 15628 catalog_manager.cc:1347] Generated new cluster ID: d65296012ace40b2a4b85c3f4db93333
I20260806 15:58:35.958627 15628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:35.964104 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:35.953593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:33231 (local address 127.0.0.1:38277)
0806 15:58:35.953729 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:35.953734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:35.954487 (+ 753us) server_negotiation.cc:407] Connection header received
0806 15:58:35.954643 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:35.954646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:35.954700 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:35.954792 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:35.955727 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:35.956954 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:35.957865 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:35.958157 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:35.958260 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:35.960190 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:35.960194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:35.960197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:35.960224 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:35.962040 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:35.962641 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:35.962645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:35.962647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:35.962744 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:35.963101 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:35.963105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:35.963107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:35.963365 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:35.963557 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:35.963820 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 15:58:35.963924 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:58:35.991153 15628 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:35.992223 15628 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:35.999221 15628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f912e8da3154fe0b92cc6aa30325ddc: Generated new TSK 0
I20260806 15:58:36.000100 15628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:36.012048 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15492
W20260806 15:58:36.027741 15469 connection.cc:535] client connection to 127.0.0.1:38277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38277): Connection reset by peer (error 104)
2026-08-06T15:58:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1643 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:36 dist-test-slave-rwrg krb5kdc[15663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:36 dist-test-slave-rwrg krb5kdc[15663](info): set up 2 sockets
Aug 06 15:58:36 dist-test-slave-rwrg krb5kdc[15663](info): commencing operation
krb5kdc: starting...
W20260806 15:58:37.341033 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:37 dist-test-slave-rwrg krb5kdc[15663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:37.475383 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:37.588069 15679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:37.588426 15679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:37.588560 15679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:37.592618 15679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:37.592703 15679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:37.592741 15679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:37.592780 15679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:37.592801 15679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:37.599313 15679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:37.600453 15679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:37.601505 15679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:37.610481 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:37.610610 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:37.610498 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:37.610849 15679 server_base.cc:1034] running on GCE node
I20260806 15:58:37.611630 15679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:37.612964 15679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:37.614192 15679 hybrid_clock.cc:648] HybridClock initialized: now 1786031917614157 us; error 50 us; skew 500 ppm
Aug 06 15:58:37 dist-test-slave-rwrg krb5kdc[15663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:37.617488 15679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:37.618402 15679 webserver.cc:460] Webserver started at http://127.0.0.1:42731/ using document root <none> and password file <none>
I20260806 15:58:37.619004 15679 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:37.619124 15679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:37.619369 15679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:37.621622 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "b5582d65f14e4ec3aa4e06a47417bdbf"
format_stamp: "Formatted at 2026-08-06 15:58:37 on dist-test-slave-rwrg"
server_key: "7f848813b2aec72bfd886b82f9658dfa"
server_key_iv: "ec37b7fb19f645587c2c138a04a605e3"
server_key_version: "encryptionkey@0"
I20260806 15:58:37.622223 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "b5582d65f14e4ec3aa4e06a47417bdbf"
format_stamp: "Formatted at 2026-08-06 15:58:37 on dist-test-slave-rwrg"
server_key: "7f848813b2aec72bfd886b82f9658dfa"
server_key_iv: "ec37b7fb19f645587c2c138a04a605e3"
server_key_version: "encryptionkey@0"
I20260806 15:58:37.626444 15679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:37.628993 15698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:37.630229 15679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:37.630378 15679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "b5582d65f14e4ec3aa4e06a47417bdbf"
format_stamp: "Formatted at 2026-08-06 15:58:37 on dist-test-slave-rwrg"
server_key: "7f848813b2aec72bfd886b82f9658dfa"
server_key_iv: "ec37b7fb19f645587c2c138a04a605e3"
server_key_version: "encryptionkey@0"
I20260806 15:58:37.630508 15679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:37.652398 15679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:37.655753 15679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:37.655975 15679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:37.663347 15679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44861
I20260806 15:58:37.663365 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44861 every 8 connection(s)
I20260806 15:58:37.664332 15679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:37.667236 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:37.672840 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15679
I20260806 15:58:37.672946 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf: Bootstrap starting.
I20260806 15:58:37.672998 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:37.673333 14204 external_mini_cluster.cc:1428] Setting key 55aea2399884ed01d7a241a8d34fa7d0
I20260806 15:58:37.675714 15791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:37.676478 15791 log.cc:824] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:37.678989 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf: No bootstrap required, opened a new log
I20260806 15:58:37.684166 15791 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260806 15:58:37.684414 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:37.674815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:34009 (local address 127.0.0.1:44861)
0806 15:58:37.675339 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:37.675357 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:37.675397 (+ 40us) server_negotiation.cc:407] Connection header received
0806 15:58:37.676162 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:37.676183 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:37.676673 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:37.677010 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:37.677404 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:37.678967 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:37.679978 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:37.680809 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:37.680993 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:37.681925 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:37.681936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:37.681958 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:37.682665 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:37.682744 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:37.682780 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:37.682931 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:37.683121 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 15:58:37.683279 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":197,"threads_started":1}
I20260806 15:58:37.684518 15791 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:37.684576 15791 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5582d65f14e4ec3aa4e06a47417bdbf, State: Initialized, Role: FOLLOWER
I20260806 15:58:37.685127 15791 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260806 15:58:37.685293 15791 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:37.685395 15791 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:37.685508 15791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:37.686626 15791 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260806 15:58:37.687053 15791 leader_election.cc:302] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5582d65f14e4ec3aa4e06a47417bdbf; no voters:
I20260806 15:58:37.687474 15791 leader_election.cc:288] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:37.687566 15800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:37.687584 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:37.687837 15800 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [term 1 LEADER]: Becoming Leader. State: Replica: b5582d65f14e4ec3aa4e06a47417bdbf, State: Running, Role: LEADER
I20260806 15:58:37.688184 15800 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260806 15:58:37.689051 15791 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:37.689538 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5582d65f14e4ec3aa4e06a47417bdbf. Latest consensus state: current_term: 1 leader_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } } }
I20260806 15:58:37.689679 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:37.690039 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5582d65f14e4ec3aa4e06a47417bdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } } }
I20260806 15:58:37.690109 15814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:37.690145 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:37.693712 15814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:37.696312 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:37.690337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:42477 (local address 127.0.0.1:44861)
0806 15:58:37.690484 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:37.690488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:37.692217 (+ 1729us) server_negotiation.cc:407] Connection header received
0806 15:58:37.692601 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:37.692605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:37.692661 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:37.692749 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:37.693018 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:37.693547 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:37.694654 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:37.694891 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:37.694994 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:37.695843 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:37.695847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:37.695849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:37.695862 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:37.695896 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:37.695906 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:37.696001 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:37.696127 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 15:58:37.696162 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 15:58:37.700275 15814 catalog_manager.cc:1347] Generated new cluster ID: bd701d77c7b044f48ca3194ced8f2621
I20260806 15:58:37.700354 15814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:37.717787 15814 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:37.718724 15814 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:37.724279 15814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5582d65f14e4ec3aa4e06a47417bdbf: Generated new TSK 0
I20260806 15:58:37.725178 15814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:37.735294 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15679
W20260806 15:58:37.746438 15655 connection.cc:535] client connection to 127.0.0.1:44861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44861): Connection reset by peer (error 104)
2026-08-06T15:58:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1718 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:37 dist-test-slave-rwrg krb5kdc[15849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:37 dist-test-slave-rwrg krb5kdc[15849](info): set up 2 sockets
Aug 06 15:58:37 dist-test-slave-rwrg krb5kdc[15849](info): commencing operation
krb5kdc: starting...
W20260806 15:58:38.993103 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:39 dist-test-slave-rwrg krb5kdc[15849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 15:58:39.120131 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:39.234052 15868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:39.234398 15868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:39.234489 15868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:39.238920 15868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:39.238994 15868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:39.239015 15868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:39.239032 15868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:39.239048 15868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:39.244481 15868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:39.245644 15868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:39.246652 15868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:39.255584 15877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:39.255584 15876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:39.255585 15879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:39.255841 15868 server_base.cc:1034] running on GCE node
I20260806 15:58:39.256457 15868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:39.257592 15868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:39.258797 15868 hybrid_clock.cc:648] HybridClock initialized: now 1786031919258753 us; error 61 us; skew 500 ppm
Aug 06 15:58:39 dist-test-slave-rwrg krb5kdc[15849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:39.261826 15868 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 15:58:39.262943 15868 webserver.cc:460] Webserver started at http://127.0.0.1:35287/ using document root <none> and password file <none>
I20260806 15:58:39.263604 15868 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:39.263710 15868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:39.263998 15868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:39.266453 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "ba218141425c4436a5122f4896ade23e"
format_stamp: "Formatted at 2026-08-06 15:58:39 on dist-test-slave-rwrg"
server_key: "429fa5089bbde5ba7b16b70a51ede196"
server_key_iv: "310f0460222e4bbe15a94f09951df049"
server_key_version: "encryptionkey@0"
I20260806 15:58:39.267135 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "ba218141425c4436a5122f4896ade23e"
format_stamp: "Formatted at 2026-08-06 15:58:39 on dist-test-slave-rwrg"
server_key: "429fa5089bbde5ba7b16b70a51ede196"
server_key_iv: "310f0460222e4bbe15a94f09951df049"
server_key_version: "encryptionkey@0"
I20260806 15:58:39.271399 15868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 15:58:39.274204 15888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:39.275540 15868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:39.275750 15868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "ba218141425c4436a5122f4896ade23e"
format_stamp: "Formatted at 2026-08-06 15:58:39 on dist-test-slave-rwrg"
server_key: "429fa5089bbde5ba7b16b70a51ede196"
server_key_iv: "310f0460222e4bbe15a94f09951df049"
server_key_version: "encryptionkey@0"
I20260806 15:58:39.275902 15868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:39.281045 15868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:39.284920 15868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:39.285202 15868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:39.293478 15983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44189 every 8 connection(s)
I20260806 15:58:39.293474 15868 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44189
I20260806 15:58:39.294584 15868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:39.296866 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 15868
I20260806 15:58:39.297049 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:39.297360 14204 external_mini_cluster.cc:1428] Setting key 68b58f22b197cf90513c9d207bc7cbbc
I20260806 15:58:39.297648 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:39.303069 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:39.298728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:43097 (local address 127.0.0.1:44189)
0806 15:58:39.299170 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:39.299190 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:39.299231 (+ 41us) server_negotiation.cc:407] Connection header received
0806 15:58:39.299996 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:39.300020 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:39.300387 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:39.300786 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:39.301664 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:39.301671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:39.301689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:39.301707 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:39.301775 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:39.301802 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:39.301943 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:39.302010 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 15:58:39.302171 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260806 15:58:39.305017 15984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e: Bootstrap starting.
I20260806 15:58:39.306135 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:39.307328 15984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:39.308040 15984 log.cc:824] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:39.310107 15984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e: No bootstrap required, opened a new log
I20260806 15:58:39.310427 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:39.308465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.237:37079 (local address 127.0.0.1:44189)
0806 15:58:39.308613 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:39.308619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:39.308886 (+ 267us) server_negotiation.cc:407] Connection header received
0806 15:58:39.309067 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:39.309070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:39.309115 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:39.309223 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:39.310007 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:39.310010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:39.310013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:39.310025 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:39.310057 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:39.310068 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:39.310166 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:39.310278 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 15:58:39.310318 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 15:58:39.313824 15984 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba218141425c4436a5122f4896ade23e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260806 15:58:39.314038 15984 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:39.314123 15984 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba218141425c4436a5122f4896ade23e, State: Initialized, Role: FOLLOWER
I20260806 15:58:39.314571 15984 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba218141425c4436a5122f4896ade23e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260806 15:58:39.314713 15984 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:39.314800 15984 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:39.314914 15984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:39.315769 15984 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba218141425c4436a5122f4896ade23e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260806 15:58:39.316090 15984 leader_election.cc:302] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba218141425c4436a5122f4896ade23e; no voters:
I20260806 15:58:39.316385 15984 leader_election.cc:288] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:39.316618 16005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:39.316916 16005 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [term 1 LEADER]: Becoming Leader. State: Replica: ba218141425c4436a5122f4896ade23e, State: Running, Role: LEADER
I20260806 15:58:39.317294 16005 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba218141425c4436a5122f4896ade23e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260806 15:58:39.317644 15984 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:39.319043 16007 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba218141425c4436a5122f4896ade23e. Latest consensus state: current_term: 1 leader_uuid: "ba218141425c4436a5122f4896ade23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba218141425c4436a5122f4896ade23e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } } }
I20260806 15:58:39.319082 16006 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba218141425c4436a5122f4896ade23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba218141425c4436a5122f4896ade23e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } } }
I20260806 15:58:39.319255 16007 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:39.319279 16006 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e [sys.catalog]: This master's current role is: LEADER
W20260806 15:58:39.322521 16032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:39.322592 16032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:39.322675 16024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:39.323346 16024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:39.328429 16024 catalog_manager.cc:1347] Generated new cluster ID: f0f289c479114c0ab28f378a150bc657
I20260806 15:58:39.328513 16024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:39.349231 16024 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:39.349615 16024 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:39.361943 16024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba218141425c4436a5122f4896ade23e: Generated new TSK 0
I20260806 15:58:39.362720 16024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:39.369127 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 15868
2026-08-06T15:58:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:39 dist-test-slave-rwrg krb5kdc[16040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:39 dist-test-slave-rwrg krb5kdc[16040](info): set up 2 sockets
Aug 06 15:58:39 dist-test-slave-rwrg krb5kdc[16040](info): commencing operation
krb5kdc: starting...
W20260806 15:58:40.653803 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:40 dist-test-slave-rwrg krb5kdc[16040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:40.783057 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:37683
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:37683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:40.894300 16057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:40.894550 16057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:40.894590 16057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:40.898499 16057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:40.898558 16057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:40.898577 16057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:40.898594 16057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:40.898609 16057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:40.903786 16057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:37683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:40.904922 16057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:40.905987 16057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:40.914750 16067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:40.914772 16070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:40.915026 16057 server_base.cc:1034] running on GCE node
W20260806 15:58:40.914774 16068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:40.915755 16057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:40.916901 16057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:40.918078 16057 hybrid_clock.cc:648] HybridClock initialized: now 1786031920918050 us; error 43 us; skew 500 ppm
Aug 06 15:58:40 dist-test-slave-rwrg krb5kdc[16040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031920, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:40.921046 16057 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:40.921903 16057 webserver.cc:460] Webserver started at http://127.13.223.62:42005/ using document root <none> and password file <none>
I20260806 15:58:40.922416 16057 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:40.922475 16057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:40.922649 16057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:40.924674 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "8a257179ffa34bd896bac1056cfd7d2e"
format_stamp: "Formatted at 2026-08-06 15:58:40 on dist-test-slave-rwrg"
server_key: "73f9df8bb226c6a42648a0191b49d48a"
server_key_iv: "346f18f54f82b3faee312db7a88684e1"
server_key_version: "encryptionkey@0"
I20260806 15:58:40.925179 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "8a257179ffa34bd896bac1056cfd7d2e"
format_stamp: "Formatted at 2026-08-06 15:58:40 on dist-test-slave-rwrg"
server_key: "73f9df8bb226c6a42648a0191b49d48a"
server_key_iv: "346f18f54f82b3faee312db7a88684e1"
server_key_version: "encryptionkey@0"
I20260806 15:58:40.928901 16057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 15:58:40.931247 16080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:40.932571 16057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:40.932724 16057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "8a257179ffa34bd896bac1056cfd7d2e"
format_stamp: "Formatted at 2026-08-06 15:58:40 on dist-test-slave-rwrg"
server_key: "73f9df8bb226c6a42648a0191b49d48a"
server_key_iv: "346f18f54f82b3faee312db7a88684e1"
server_key_version: "encryptionkey@0"
I20260806 15:58:40.932843 16057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:40.957226 16057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:40.960237 16057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:40.960563 16057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:40.967821 16057 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:37683
I20260806 15:58:40.967837 16160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:37683 every 8 connection(s)
I20260806 15:58:40.968796 16057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:40.970338 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16057
I20260806 15:58:40.970500 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:40.970849 14204 external_mini_cluster.cc:1428] Setting key 59d3f5a1980cec8e0c628a333163fea0
I20260806 15:58:40.972231 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:40.979090 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e: Bootstrap starting.
Aug 06 15:58:40 dist-test-slave-rwrg krb5kdc[16040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:40.981596 16162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:40.982563 16162 log.cc:824] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:40.984941 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e: No bootstrap required, opened a new log
I20260806 15:58:40.986438 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:40.972203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33511 (local address 127.13.223.62:37683)
0806 15:58:40.972698 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:40.972718 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:40.972763 (+ 45us) server_negotiation.cc:407] Connection header received
0806 15:58:40.973611 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:40.973632 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:40.974037 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:40.974403 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:40.975097 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:40.976609 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:40.977458 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:40.978202 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:40.978363 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:40.981018 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:40.981025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:40.981040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:40.981070 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:40.983076 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:40.983724 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:40.983730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:40.983736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:40.983866 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:40.984227 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:40.984230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:40.984232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:40.984752 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:40.984953 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:40.985296 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 15:58:40.985522 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":143,"threads_started":1}
I20260806 15:58:40.989243 16162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a257179ffa34bd896bac1056cfd7d2e" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37683 } }
I20260806 15:58:40.989566 16162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:40.989650 16162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a257179ffa34bd896bac1056cfd7d2e, State: Initialized, Role: FOLLOWER
I20260806 15:58:40.989746 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:40.990093 16162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a257179ffa34bd896bac1056cfd7d2e" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37683 } }
I20260806 15:58:40.990272 16162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:40.990334 16162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:40.990432 16162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:40.991601 16162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a257179ffa34bd896bac1056cfd7d2e" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37683 } }
I20260806 15:58:40.991991 16162 leader_election.cc:302] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a257179ffa34bd896bac1056cfd7d2e; no voters:
I20260806 15:58:40.992370 16162 leader_election.cc:288] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:40.992484 16178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:40.992833 16178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [term 1 LEADER]: Becoming Leader. State: Replica: 8a257179ffa34bd896bac1056cfd7d2e, State: Running, Role: LEADER
I20260806 15:58:40.993199 16178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a257179ffa34bd896bac1056cfd7d2e" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37683 } }
I20260806 15:58:40.993784 16162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:40.994421 16178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a257179ffa34bd896bac1056cfd7d2e. Latest consensus state: current_term: 1 leader_uuid: "8a257179ffa34bd896bac1056cfd7d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a257179ffa34bd896bac1056cfd7d2e" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37683 } } }
I20260806 15:58:40.994626 16178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:40.994853 16181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a257179ffa34bd896bac1056cfd7d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a257179ffa34bd896bac1056cfd7d2e" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 37683 } } }
I20260806 15:58:40.995021 16181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:40.995124 16194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:40.998777 16194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:41.003067 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:40.992116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59851 (local address 127.13.223.62:37683)
0806 15:58:40.992233 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:40.992239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:40.992529 (+ 290us) server_negotiation.cc:407] Connection header received
0806 15:58:40.992741 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:40.992744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:40.992786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:40.992831 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:40.993740 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:40.994447 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:40.995667 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:40.995992 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:40.996050 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:40.998956 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:40.998961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:40.998963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:40.998992 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:41.000982 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:41.001574 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:41.001578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:41.001580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:41.001683 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:41.002075 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:41.002079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:41.002081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:41.002277 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:41.002453 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:41.002785 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 15:58:41.002886 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 15:58:41.006000 16194 catalog_manager.cc:1347] Generated new cluster ID: 08896a295afa4b519d089f10d5bed475
I20260806 15:58:41.006083 16194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:41.020267 16194 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:41.021358 16194 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:41.031041 16194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a257179ffa34bd896bac1056cfd7d2e: Generated new TSK 0
I20260806 15:58:41.031869 16194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:41.040954 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16057
W20260806 15:58:41.058063 16038 connection.cc:535] client connection to 127.13.223.62:37683 recv error: Network error: failed to read from TLS socket (remote: 127.13.223.62:37683): Connection reset by peer (error 104)
2026-08-06T15:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1669 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:41 dist-test-slave-rwrg krb5kdc[16216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:41 dist-test-slave-rwrg krb5kdc[16216](info): set up 2 sockets
Aug 06 15:58:41 dist-test-slave-rwrg krb5kdc[16216](info): commencing operation
krb5kdc: starting...
W20260806 15:58:42.297672 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:42 dist-test-slave-rwrg krb5kdc[16216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:42.425660 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39037
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:39037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:42.537269 16234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:42.537537 16234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:42.537578 16234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:42.541504 16234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:42.541556 16234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:42.541575 16234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:42.541591 16234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:42.541605 16234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:42.546779 16234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:39037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:42.547784 16234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:42.548776 16234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:42.557360 16242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:42.557408 16243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:42.557369 16248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:42.557559 16234 server_base.cc:1034] running on GCE node
I20260806 15:58:42.558218 16234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:42.559309 16234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:42.560484 16234 hybrid_clock.cc:648] HybridClock initialized: now 1786031922560464 us; error 32 us; skew 500 ppm
Aug 06 15:58:42 dist-test-slave-rwrg krb5kdc[16216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:42.563362 16234 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:42.564273 16234 webserver.cc:460] Webserver started at http://127.13.223.62:45723/ using document root <none> and password file <none>
I20260806 15:58:42.564908 16234 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:42.564970 16234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:42.565191 16234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:42.567157 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "c7e5c2c24f8f4a29aff94df53a53041d"
format_stamp: "Formatted at 2026-08-06 15:58:42 on dist-test-slave-rwrg"
server_key: "7f4b2529415a21fc20336338adcdb2f2"
server_key_iv: "52b1709fc3162a77a9b7b6ebf793ec18"
server_key_version: "encryptionkey@0"
I20260806 15:58:42.567756 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "c7e5c2c24f8f4a29aff94df53a53041d"
format_stamp: "Formatted at 2026-08-06 15:58:42 on dist-test-slave-rwrg"
server_key: "7f4b2529415a21fc20336338adcdb2f2"
server_key_iv: "52b1709fc3162a77a9b7b6ebf793ec18"
server_key_version: "encryptionkey@0"
I20260806 15:58:42.571609 16234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 15:58:42.574039 16255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:42.575309 16234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 15:58:42.575546 16234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "c7e5c2c24f8f4a29aff94df53a53041d"
format_stamp: "Formatted at 2026-08-06 15:58:42 on dist-test-slave-rwrg"
server_key: "7f4b2529415a21fc20336338adcdb2f2"
server_key_iv: "52b1709fc3162a77a9b7b6ebf793ec18"
server_key_version: "encryptionkey@0"
I20260806 15:58:42.575654 16234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:42.594980 16234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:42.598389 16234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:42.598604 16234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:42.606089 16234 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:39037
I20260806 15:58:42.606114 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:39037 every 8 connection(s)
I20260806 15:58:42.607085 16234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:42.609933 16353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:42.612215 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16234
I20260806 15:58:42.612378 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:42.612746 14204 external_mini_cluster.cc:1428] Setting key 55610f036b700bd60a19491287e798d8
I20260806 15:58:42.616701 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d: Bootstrap starting.
I20260806 15:58:42.619050 16353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:42.619812 16353 log.cc:824] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:42.621830 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d: No bootstrap required, opened a new log
I20260806 15:58:42.622879 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:42.614678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54847 (local address 127.13.223.62:39037)
0806 15:58:42.615625 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:42.615642 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:42.615677 (+ 35us) server_negotiation.cc:407] Connection header received
0806 15:58:42.616294 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:42.616309 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:42.616735 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:42.617105 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:42.617516 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:42.618635 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:42.619594 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:42.620118 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:42.620293 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:42.621213 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:42.621218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:42.621231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:42.621625 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:42.621678 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:42.621699 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:42.621839 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:42.622037 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 15:58:42.622179 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":157,"threads_started":1}
I20260806 15:58:42.626091 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:42.626233 16353 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39037 } }
I20260806 15:58:42.626477 16353 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:42.626564 16353 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e5c2c24f8f4a29aff94df53a53041d, State: Initialized, Role: FOLLOWER
I20260806 15:58:42.627066 16353 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39037 } }
I20260806 15:58:42.627223 16353 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:42.627310 16353 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:42.627424 16353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:42.628621 16353 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39037 } }
I20260806 15:58:42.628968 16353 leader_election.cc:302] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7e5c2c24f8f4a29aff94df53a53041d; no voters:
I20260806 15:58:42.629354 16353 leader_election.cc:288] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:42.629426 16369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:42.629642 16369 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [term 1 LEADER]: Becoming Leader. State: Replica: c7e5c2c24f8f4a29aff94df53a53041d, State: Running, Role: LEADER
I20260806 15:58:42.630026 16369 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39037 } }
I20260806 15:58:42.630759 16353 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:42.631834 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7e5c2c24f8f4a29aff94df53a53041d. Latest consensus state: current_term: 1 leader_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39037 } } }
I20260806 15:58:42.631979 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:42.632165 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e5c2c24f8f4a29aff94df53a53041d" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39037 } } }
I20260806 15:58:42.632614 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:42.632745 16379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:42.633358 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:42.628476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40479 (local address 127.13.223.62:39037)
0806 15:58:42.628690 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:42.628694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:42.628806 (+ 112us) server_negotiation.cc:407] Connection header received
0806 15:58:42.628964 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:42.628967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:42.629019 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:42.629104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:42.629421 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:42.630135 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:42.631093 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:42.631448 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:42.631581 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:42.632786 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:42.632790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:42.632793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:42.632806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:42.632837 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:42.632854 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:42.632949 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:42.633195 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 15:58:42.633232 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260806 15:58:42.636175 16379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:42.642107 16379 catalog_manager.cc:1347] Generated new cluster ID: cd40e672ef924b8796115ae1ac3b68bd
I20260806 15:58:42.642189 16379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:42.660137 16379 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:42.661633 16379 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:42.678138 16379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7e5c2c24f8f4a29aff94df53a53041d: Generated new TSK 0
I20260806 15:58:42.679075 16379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:42.683794 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16234
W20260806 15:58:42.699959 16213 connection.cc:535] client connection to 127.13.223.62:39037 recv error: Network error: failed to read from TLS socket (remote: 127.13.223.62:39037): Connection reset by peer (error 104)
2026-08-06T15:58:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:42 dist-test-slave-rwrg krb5kdc[16404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:42 dist-test-slave-rwrg krb5kdc[16404](info): set up 2 sockets
Aug 06 15:58:42 dist-test-slave-rwrg krb5kdc[16404](info): commencing operation
krb5kdc: starting...
W20260806 15:58:43.971803 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:44 dist-test-slave-rwrg krb5kdc[16404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:44.101279 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39875
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:39875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:44.212991 16422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:44.213312 16422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:44.213387 16422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:44.217458 16422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:44.217542 16422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:44.217579 16422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:44.217638 16422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:44.217675 16422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:44.223256 16422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:39875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:39875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:44.224306 16422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:44.225368 16422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:44.234069 16428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:44.234126 16422 server_base.cc:1034] running on GCE node
W20260806 15:58:44.234076 16433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:44.234057 16430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:44.234892 16422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:44.235955 16422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:44.237155 16422 hybrid_clock.cc:648] HybridClock initialized: now 1786031924237141 us; error 26 us; skew 500 ppm
Aug 06 15:58:44 dist-test-slave-rwrg krb5kdc[16404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031924, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:44.239873 16422 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:44.240862 16422 webserver.cc:460] Webserver started at http://127.13.223.62:36469/ using document root <none> and password file <none>
I20260806 15:58:44.241416 16422 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:44.241509 16422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:44.241822 16422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:44.243860 16422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "b4ae99e43bda4d1c98ae8e29912677b0"
format_stamp: "Formatted at 2026-08-06 15:58:44 on dist-test-slave-rwrg"
server_key: "a972555c385870c9675530f4b7f2a7ad"
server_key_iv: "2cd3eab3673a8b79f84266861aaaa3e0"
server_key_version: "encryptionkey@0"
I20260806 15:58:44.244432 16422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "b4ae99e43bda4d1c98ae8e29912677b0"
format_stamp: "Formatted at 2026-08-06 15:58:44 on dist-test-slave-rwrg"
server_key: "a972555c385870c9675530f4b7f2a7ad"
server_key_iv: "2cd3eab3673a8b79f84266861aaaa3e0"
server_key_version: "encryptionkey@0"
I20260806 15:58:44.248059 16422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:44.250638 16448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:44.251897 16422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 15:58:44.252056 16422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "b4ae99e43bda4d1c98ae8e29912677b0"
format_stamp: "Formatted at 2026-08-06 15:58:44 on dist-test-slave-rwrg"
server_key: "a972555c385870c9675530f4b7f2a7ad"
server_key_iv: "2cd3eab3673a8b79f84266861aaaa3e0"
server_key_version: "encryptionkey@0"
I20260806 15:58:44.252170 16422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:44.256745 16422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:44.259716 16422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:44.259922 16422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:44.267328 16422 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:39875
I20260806 15:58:44.267352 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:39875 every 8 connection(s)
I20260806 15:58:44.268340 16422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:44.271144 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:44.276643 16542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0: Bootstrap starting.
I20260806 15:58:44.278280 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16422
I20260806 15:58:44.278426 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:44.278744 14204 external_mini_cluster.cc:1428] Setting key 83587f7612725ae34d7f1ade9dd88d87
I20260806 15:58:44.279420 16542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:44.280383 16542 log.cc:824] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:44.282433 16542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0: No bootstrap required, opened a new log
I20260806 15:58:44.283850 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:44.279947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53115 (local address 127.13.223.62:39875)
0806 15:58:44.280396 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:44.280411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:44.280447 (+ 36us) server_negotiation.cc:407] Connection header received
0806 15:58:44.281210 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:44.281228 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:44.281498 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:44.281834 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:44.282672 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:44.282676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:44.282688 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:44.282699 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:44.282753 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:44.282771 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:44.282889 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:44.282995 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 15:58:44.283170 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260806 15:58:44.286721 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:44.286633 16542 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39875 } }
I20260806 15:58:44.286919 16542 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:44.287005 16542 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ae99e43bda4d1c98ae8e29912677b0, State: Initialized, Role: FOLLOWER
I20260806 15:58:44.287410 16542 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39875 } }
I20260806 15:58:44.287557 16542 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:44.287649 16542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:44.287748 16542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:44.288805 16542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39875 } }
I20260806 15:58:44.289194 16542 leader_election.cc:302] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ae99e43bda4d1c98ae8e29912677b0; no voters:
I20260806 15:58:44.289615 16542 leader_election.cc:288] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:44.289710 16559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:44.289987 16559 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [term 1 LEADER]: Becoming Leader. State: Replica: b4ae99e43bda4d1c98ae8e29912677b0, State: Running, Role: LEADER
I20260806 15:58:44.290356 16559 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39875 } }
I20260806 15:58:44.290843 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:44.289045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33503 (local address 127.13.223.62:39875)
0806 15:58:44.289150 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:44.289155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:44.289375 (+ 220us) server_negotiation.cc:407] Connection header received
0806 15:58:44.289493 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:44.289495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:44.289524 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:44.289614 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:44.290359 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:44.290363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:44.290369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:44.290384 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:44.290411 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:44.290424 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:44.290510 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:44.290670 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 15:58:44.290709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 15:58:44.291029 16542 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:44.293181 16563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ae99e43bda4d1c98ae8e29912677b0. Latest consensus state: current_term: 1 leader_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39875 } } }
I20260806 15:58:44.293313 16563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:44.293473 16560 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ae99e43bda4d1c98ae8e29912677b0" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 39875 } } }
I20260806 15:58:44.293743 16590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:44.294137 16560 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:44.296798 16590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:44.302309 16590 catalog_manager.cc:1347] Generated new cluster ID: 81a6d7c667184e3f96fa3c54d507562c
I20260806 15:58:44.302386 16590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:44.320410 16590 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:44.320811 16590 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:44.328692 16590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ae99e43bda4d1c98ae8e29912677b0: Generated new TSK 0
I20260806 15:58:44.329365 16590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:44.335667 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16422
2026-08-06T15:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1656 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:44 dist-test-slave-rwrg krb5kdc[16597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:44 dist-test-slave-rwrg krb5kdc[16597](info): set up 2 sockets
Aug 06 15:58:44 dist-test-slave-rwrg krb5kdc[16597](info): commencing operation
krb5kdc: starting...
W20260806 15:58:45.640203 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:45 dist-test-slave-rwrg krb5kdc[16597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:45.769716 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40171
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:45.883603 16617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:45.883909 16617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:45.883991 16617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:45.887904 16617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:45.888002 16617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:45.888041 16617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:45.888095 16617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:45.888129 16617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:45.893491 16617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:45.894639 16617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:45.895670 16617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:45.904337 16626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:45.904347 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:45.904351 16625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:45.904789 16617 server_base.cc:1034] running on GCE node
I20260806 15:58:45.905361 16617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:45.906427 16617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:45.907585 16617 hybrid_clock.cc:648] HybridClock initialized: now 1786031925907560 us; error 40 us; skew 500 ppm
Aug 06 15:58:45 dist-test-slave-rwrg krb5kdc[16597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031925, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:45.910396 16617 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:45.911304 16617 webserver.cc:460] Webserver started at http://127.13.223.62:33963/ using document root <none> and password file <none>
I20260806 15:58:45.911872 16617 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:45.911974 16617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:45.912217 16617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:45.914294 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "8944c75e1e5c45dcaadac5ab65fafeb7"
format_stamp: "Formatted at 2026-08-06 15:58:45 on dist-test-slave-rwrg"
server_key: "928113261d5872e823fcc0aabf9d88e7"
server_key_iv: "6c635cb084030a72a809ef579cfd41d9"
server_key_version: "encryptionkey@0"
I20260806 15:58:45.914911 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "8944c75e1e5c45dcaadac5ab65fafeb7"
format_stamp: "Formatted at 2026-08-06 15:58:45 on dist-test-slave-rwrg"
server_key: "928113261d5872e823fcc0aabf9d88e7"
server_key_iv: "6c635cb084030a72a809ef579cfd41d9"
server_key_version: "encryptionkey@0"
I20260806 15:58:45.918859 16617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 15:58:45.921272 16638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:45.922503 16617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260806 15:58:45.922660 16617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "8944c75e1e5c45dcaadac5ab65fafeb7"
format_stamp: "Formatted at 2026-08-06 15:58:45 on dist-test-slave-rwrg"
server_key: "928113261d5872e823fcc0aabf9d88e7"
server_key_iv: "6c635cb084030a72a809ef579cfd41d9"
server_key_version: "encryptionkey@0"
I20260806 15:58:45.922776 16617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:45.945395 16617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:45.948359 16617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:45.948611 16617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:45.956004 16617 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:40171
I20260806 15:58:45.956018 16729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:40171 every 8 connection(s)
I20260806 15:58:45.957007 16617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:45.959800 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:45.965659 16730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7: Bootstrap starting.
I20260806 15:58:45.966923 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16617
I20260806 15:58:45.967090 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:45.967360 14204 external_mini_cluster.cc:1428] Setting key b8ab390c377258c209d6ea8095b7a2cd
I20260806 15:58:45.968432 16730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:45.969460 16730 log.cc:824] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:45.971736 16730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7: No bootstrap required, opened a new log
I20260806 15:58:45.976225 16730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40171 } }
I20260806 15:58:45.976594 16730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:45.976672 16730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8944c75e1e5c45dcaadac5ab65fafeb7, State: Initialized, Role: FOLLOWER
Aug 06 15:58:45 dist-test-slave-rwrg krb5kdc[16597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:45.977216 16730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40171 } }
I20260806 15:58:45.977393 16730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:45.977489 16730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:45.977624 16730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:45.978950 16730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40171 } }
I20260806 15:58:45.979318 16730 leader_election.cc:302] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8944c75e1e5c45dcaadac5ab65fafeb7; no voters:
I20260806 15:58:45.979738 16730 leader_election.cc:288] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:45.979861 16736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:45.980168 16736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [term 1 LEADER]: Becoming Leader. State: Replica: 8944c75e1e5c45dcaadac5ab65fafeb7, State: Running, Role: LEADER
I20260806 15:58:45.980547 16736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40171 } }
I20260806 15:58:45.981307 16730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:45.982293 16737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40171 } } }
I20260806 15:58:45.982412 16737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:45.983829 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8944c75e1e5c45dcaadac5ab65fafeb7. Latest consensus state: current_term: 1 leader_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8944c75e1e5c45dcaadac5ab65fafeb7" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 40171 } } }
I20260806 15:58:45.983925 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:45.984194 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:45.968534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40375 (local address 127.13.223.62:40171)
0806 15:58:45.969028 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:45.969050 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:45.969096 (+ 46us) server_negotiation.cc:407] Connection header received
0806 15:58:45.969998 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:45.970021 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:45.970428 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:45.970814 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:45.971464 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:45.972992 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:45.973862 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:45.974648 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:45.974881 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:45.977704 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:45.977711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:45.977746 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:45.977782 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:45.979694 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:45.980309 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:45.980315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:45.980321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:45.980470 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:45.980892 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:45.980895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:45.980897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:45.981403 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:45.981628 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:45.982845 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0806 15:58:45.983143 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":151,"threads_started":1}
W20260806 15:58:45.987090 16758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:45.987174 16758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:45.987244 16749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:45.987512 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:45.988044 16749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:45.994553 16749 catalog_manager.cc:1347] Generated new cluster ID: 0d2fe5ac89164a30989ce89663d1629c
I20260806 15:58:45.994621 16749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:45.999917 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:45.989621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34429 (local address 127.13.223.62:40171)
0806 15:58:45.989732 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:45.989737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:45.989972 (+ 235us) server_negotiation.cc:407] Connection header received
0806 15:58:45.990116 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:45.990124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:45.990161 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:45.990250 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:45.990920 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:45.991437 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:45.992441 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:45.992759 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:45.992869 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:45.996615 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:45.996619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:45.996622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:45.996650 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:45.998169 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:45.998642 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:45.998645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:45.998649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:45.998741 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:45.999030 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:45.999035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:45.999037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:45.999218 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:45.999355 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:45.999601 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 15:58:45.999733 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 15:58:46.008365 16749 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:46.009833 16749 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:46.017359 16749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8944c75e1e5c45dcaadac5ab65fafeb7: Generated new TSK 0
I20260806 15:58:46.018163 16749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:46.023530 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16617
2026-08-06T15:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1684 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:46 dist-test-slave-rwrg krb5kdc[16775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:46 dist-test-slave-rwrg krb5kdc[16775](info): set up 2 sockets
Aug 06 15:58:46 dist-test-slave-rwrg krb5kdc[16775](info): commencing operation
krb5kdc: starting...
W20260806 15:58:47.277523 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:47 dist-test-slave-rwrg krb5kdc[16775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:47.409633 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33303
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:47.523372 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:47.523670 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:47.523715 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:47.527598 16800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:47.527650 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:47.527669 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:47.527685 16800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:47.527700 16800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:47.532963 16800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:33303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:47.533926 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:47.534865 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:47.543398 16809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:47.543387 16806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:47.543365 16807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:47.543699 16800 server_base.cc:1034] running on GCE node
I20260806 15:58:47.544368 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:47.545528 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:47.546730 16800 hybrid_clock.cc:648] HybridClock initialized: now 1786031927546680 us; error 61 us; skew 500 ppm
Aug 06 15:58:47 dist-test-slave-rwrg krb5kdc[16775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031927, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:47.549723 16800 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:47.550625 16800 webserver.cc:460] Webserver started at http://127.13.223.62:36693/ using document root <none> and password file <none>
I20260806 15:58:47.551190 16800 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:47.551286 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:47.551522 16800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:47.553639 16800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "26ed299ba27548bd8e44465c52eb4e7b"
format_stamp: "Formatted at 2026-08-06 15:58:47 on dist-test-slave-rwrg"
server_key: "bea15724be0091381bafae70d6b424fe"
server_key_iv: "a6f65c388f51cc9b407b338d9f85cbea"
server_key_version: "encryptionkey@0"
I20260806 15:58:47.554258 16800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "26ed299ba27548bd8e44465c52eb4e7b"
format_stamp: "Formatted at 2026-08-06 15:58:47 on dist-test-slave-rwrg"
server_key: "bea15724be0091381bafae70d6b424fe"
server_key_iv: "a6f65c388f51cc9b407b338d9f85cbea"
server_key_version: "encryptionkey@0"
I20260806 15:58:47.558223 16800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:47.560835 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:47.561995 16800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:47.562145 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "26ed299ba27548bd8e44465c52eb4e7b"
format_stamp: "Formatted at 2026-08-06 15:58:47 on dist-test-slave-rwrg"
server_key: "bea15724be0091381bafae70d6b424fe"
server_key_iv: "a6f65c388f51cc9b407b338d9f85cbea"
server_key_version: "encryptionkey@0"
I20260806 15:58:47.562255 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:47.579627 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:47.582662 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:47.582885 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:47.590046 16800 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:33303
I20260806 15:58:47.590122 16907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:33303 every 8 connection(s)
I20260806 15:58:47.591084 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:47.593915 16908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:47.596753 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16800
I20260806 15:58:47.596918 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:47.597211 14204 external_mini_cluster.cc:1428] Setting key 948b7d0e942abb123185845afc9e0ed4
I20260806 15:58:47.599929 16908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b: Bootstrap starting.
I20260806 15:58:47.602617 16908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:47.603375 16908 log.cc:824] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:47.605692 16908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b: No bootstrap required, opened a new log
I20260806 15:58:47.607182 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:47.598662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37375 (local address 127.13.223.62:33303)
0806 15:58:47.599084 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:47.599113 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:47.599155 (+ 42us) server_negotiation.cc:407] Connection header received
0806 15:58:47.599930 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:47.599953 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:47.600392 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:47.600773 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:47.601212 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:47.602677 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:47.603672 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:47.604246 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:47.604404 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:47.605253 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:47.605260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:47.605276 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:47.605763 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:47.605828 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:47.605858 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:47.605955 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:47.606071 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 15:58:47.606274 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260806 15:58:47.609349 16908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed299ba27548bd8e44465c52eb4e7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33303 } }
I20260806 15:58:47.609638 16908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:47.609709 16908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ed299ba27548bd8e44465c52eb4e7b, State: Initialized, Role: FOLLOWER
I20260806 15:58:47.610172 16908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed299ba27548bd8e44465c52eb4e7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33303 } }
I20260806 15:58:47.610337 16908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:47.610441 16908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:47.610553 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:47.610558 16908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:47.611460 16908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed299ba27548bd8e44465c52eb4e7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33303 } }
I20260806 15:58:47.611828 16908 leader_election.cc:302] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26ed299ba27548bd8e44465c52eb4e7b; no voters:
I20260806 15:58:47.612134 16908 leader_election.cc:288] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:47.612277 16924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:47.612573 16924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [term 1 LEADER]: Becoming Leader. State: Replica: 26ed299ba27548bd8e44465c52eb4e7b, State: Running, Role: LEADER
I20260806 15:58:47.612943 16924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed299ba27548bd8e44465c52eb4e7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33303 } }
I20260806 15:58:47.613533 16908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:47.614895 16926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26ed299ba27548bd8e44465c52eb4e7b. Latest consensus state: current_term: 1 leader_uuid: "26ed299ba27548bd8e44465c52eb4e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed299ba27548bd8e44465c52eb4e7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33303 } } }
I20260806 15:58:47.614885 16925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26ed299ba27548bd8e44465c52eb4e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ed299ba27548bd8e44465c52eb4e7b" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33303 } } }
I20260806 15:58:47.615046 16925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:47.615046 16926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:47.617691 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:47.612562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38733 (local address 127.13.223.62:33303)
0806 15:58:47.612659 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:47.612663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:47.613129 (+ 466us) server_negotiation.cc:407] Connection header received
0806 15:58:47.613310 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:47.613314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:47.613360 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:47.613444 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:47.613745 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:47.614552 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:47.615782 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:47.616070 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:47.616202 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:47.617020 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:47.617023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:47.617026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:47.617038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:47.617073 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:47.617086 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:47.617167 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:47.617466 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 15:58:47.617523 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 15:58:47.618677 16943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:47.618764 16943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:47.618841 16939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:47.619691 16939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:47.625036 16939 catalog_manager.cc:1347] Generated new cluster ID: 4ba06309a1214b6ebcf478bcb33d1b7b
I20260806 15:58:47.625101 16939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:47.663089 16939 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:47.664088 16939 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:47.671677 16939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26ed299ba27548bd8e44465c52eb4e7b: Generated new TSK 0
I20260806 15:58:47.672551 16939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:47.686607 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16800
2026-08-06T15:58:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:47 dist-test-slave-rwrg krb5kdc[16950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:47 dist-test-slave-rwrg krb5kdc[16950](info): set up 2 sockets
Aug 06 15:58:47 dist-test-slave-rwrg krb5kdc[16950](info): commencing operation
krb5kdc: starting...
W20260806 15:58:48.955493 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:49 dist-test-slave-rwrg krb5kdc[16950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:49.085474 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33041
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:33041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:49.199544 16969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:49.199808 16969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:49.199849 16969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:49.203910 16969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:49.203976 16969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:49.203996 16969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:49.204012 16969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:49.204027 16969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:49.209430 16969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:33041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:49.210464 16969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:49.211424 16969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:58:49.220090 16969 server_base.cc:1034] running on GCE node
W20260806 15:58:49.220029 16983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:49.219991 16985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:49.219991 16982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:49.220852 16969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:49.221953 16969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:49.223150 16969 hybrid_clock.cc:648] HybridClock initialized: now 1786031929223136 us; error 33 us; skew 500 ppm
Aug 06 15:58:49 dist-test-slave-rwrg krb5kdc[16950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:49.226126 16969 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:49.226938 16969 webserver.cc:460] Webserver started at http://127.13.223.62:42513/ using document root <none> and password file <none>
I20260806 15:58:49.227430 16969 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:49.227486 16969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:49.227643 16969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:49.229687 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "0f69717125b948dcbe47af205175e474"
format_stamp: "Formatted at 2026-08-06 15:58:49 on dist-test-slave-rwrg"
server_key: "47b4d29f628f8482b89d15927a90263c"
server_key_iv: "ed4efb4d4482c6dcf6230b9c991733c6"
server_key_version: "encryptionkey@0"
I20260806 15:58:49.230218 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "0f69717125b948dcbe47af205175e474"
format_stamp: "Formatted at 2026-08-06 15:58:49 on dist-test-slave-rwrg"
server_key: "47b4d29f628f8482b89d15927a90263c"
server_key_iv: "ed4efb4d4482c6dcf6230b9c991733c6"
server_key_version: "encryptionkey@0"
I20260806 15:58:49.233857 16969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:58:49.236416 16998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:49.237792 16969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:49.237905 16969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "0f69717125b948dcbe47af205175e474"
format_stamp: "Formatted at 2026-08-06 15:58:49 on dist-test-slave-rwrg"
server_key: "47b4d29f628f8482b89d15927a90263c"
server_key_iv: "ed4efb4d4482c6dcf6230b9c991733c6"
server_key_version: "encryptionkey@0"
I20260806 15:58:49.237988 16969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:49.242432 16969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:49.245687 16969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:49.245883 16969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:49.253161 16969 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:33041
I20260806 15:58:49.253248 17077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:33041 every 8 connection(s)
I20260806 15:58:49.254163 16969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:49.257066 17078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:49.262686 17078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474: Bootstrap starting.
I20260806 15:58:49.263084 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 16969
I20260806 15:58:49.263231 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:49.263504 14204 external_mini_cluster.cc:1428] Setting key 6d9ef8b548a5aea892b73fb850ba0c16
I20260806 15:58:49.265746 17078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:49.266753 17078 log.cc:824] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:49.269053 17078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474: No bootstrap required, opened a new log
I20260806 15:58:49.269749 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:49.264968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48333 (local address 127.13.223.62:33041)
0806 15:58:49.265679 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:49.265702 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:49.265753 (+ 51us) server_negotiation.cc:407] Connection header received
0806 15:58:49.266725 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:49.266751 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:49.267138 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:49.267552 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:49.268370 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:49.268383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:49.268395 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:49.268407 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:49.268467 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:49.268489 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:49.268655 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:49.268770 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 15:58:49.268935 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":143,"threads_started":1}
I20260806 15:58:49.272634 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:49.272838 17078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f69717125b948dcbe47af205175e474" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33041 } }
I20260806 15:58:49.273072 17078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 15:58:49.273173 17078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f69717125b948dcbe47af205175e474, State: Initialized, Role: FOLLOWER
I20260806 15:58:49.273583 17078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f69717125b948dcbe47af205175e474" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33041 } }
I20260806 15:58:49.273733 17078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 15:58:49.273818 17078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 15:58:49.273938 17078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:49.274823 17078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f69717125b948dcbe47af205175e474" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33041 } }
I20260806 15:58:49.275255 17078 leader_election.cc:302] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f69717125b948dcbe47af205175e474; no voters:
I20260806 15:58:49.275583 17078 leader_election.cc:288] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 15:58:49.275990 17095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:49.276257 17095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [term 1 LEADER]: Becoming Leader. State: Replica: 0f69717125b948dcbe47af205175e474, State: Running, Role: LEADER
I20260806 15:58:49.276640 17095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f69717125b948dcbe47af205175e474" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33041 } }
I20260806 15:58:49.276908 17078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:49.277099 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:49.275052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50065 (local address 127.13.223.62:33041)
0806 15:58:49.275183 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:49.275188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:49.275379 (+ 191us) server_negotiation.cc:407] Connection header received
0806 15:58:49.275523 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:49.275526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:49.275560 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:49.275632 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:49.276481 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:49.276485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:49.276487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 15:58:49.276531 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:49.276558 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 15:58:49.276573 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:49.276678 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:49.276953 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 15:58:49.276995 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 15:58:49.278257 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f69717125b948dcbe47af205175e474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f69717125b948dcbe47af205175e474" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33041 } } }
I20260806 15:58:49.278380 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:49.278335 17096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f69717125b948dcbe47af205175e474. Latest consensus state: current_term: 1 leader_uuid: "0f69717125b948dcbe47af205175e474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f69717125b948dcbe47af205175e474" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 33041 } } }
I20260806 15:58:49.278482 17096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474 [sys.catalog]: This master's current role is: LEADER
W20260806 15:58:49.282022 17115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:49.282115 17115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:49.282225 17109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:49.282949 17109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:49.288566 17109 catalog_manager.cc:1347] Generated new cluster ID: 4191ac8a2bbf4cbda0397b9ee96a5afe
I20260806 15:58:49.288645 17109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:49.309444 17109 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:49.309893 17109 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:49.317529 17109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f69717125b948dcbe47af205175e474: Generated new TSK 0
I20260806 15:58:49.318252 17109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 15:58:49.325529 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 16969
2026-08-06T15:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1634 ms)
[----------] 18 tests from AuthTokenIssuingTest (29936 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:49 dist-test-slave-rwrg krb5kdc[17125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:49 dist-test-slave-rwrg krb5kdc[17125](info): set up 2 sockets
Aug 06 15:58:49 dist-test-slave-rwrg krb5kdc[17125](info): commencing operation
krb5kdc: starting...
W20260806 15:58:50.592847 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:50 dist-test-slave-rwrg krb5kdc[17125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:50.720081 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:36187
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:36187,127.13.223.61:37509,127.13.223.60:44591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:50.835191 17145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 15:58:50.835579 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:50.835675 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:50.835754 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:50.840044 17145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:50.840132 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:50.840170 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:50.840209 17145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:50.840245 17145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:50.845602 17145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:36187,127.13.223.61:37509,127.13.223.60:44591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:50.846717 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:50.847776 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:50.856406 17156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:50.856379 17150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:50.856760 17145 server_base.cc:1034] running on GCE node
W20260806 15:58:50.856383 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:50.857364 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:50.858552 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:50.859759 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786031930859726 us; error 50 us; skew 500 ppm
Aug 06 15:58:50 dist-test-slave-rwrg krb5kdc[17125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:50.866227 17145 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:50.867184 17145 webserver.cc:460] Webserver started at http://127.13.223.62:34931/ using document root <none> and password file <none>
I20260806 15:58:50.867760 17145 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:50.867854 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:50.868108 17145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:50.870296 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "7be12b4b51ab4939a30f05b1cac93e24"
format_stamp: "Formatted at 2026-08-06 15:58:50 on dist-test-slave-rwrg"
server_key: "c4f79a02c1905f85ef100ba850dfe866"
server_key_iv: "bffcf34da075563b7d4f3a60c0399622"
server_key_version: "encryptionkey@0"
I20260806 15:58:50.870885 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "7be12b4b51ab4939a30f05b1cac93e24"
format_stamp: "Formatted at 2026-08-06 15:58:50 on dist-test-slave-rwrg"
server_key: "c4f79a02c1905f85ef100ba850dfe866"
server_key_iv: "bffcf34da075563b7d4f3a60c0399622"
server_key_version: "encryptionkey@0"
I20260806 15:58:50.874771 17145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 15:58:50.877238 17168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:50.878441 17145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:58:50.878599 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "7be12b4b51ab4939a30f05b1cac93e24"
format_stamp: "Formatted at 2026-08-06 15:58:50 on dist-test-slave-rwrg"
server_key: "c4f79a02c1905f85ef100ba850dfe866"
server_key_iv: "bffcf34da075563b7d4f3a60c0399622"
server_key_version: "encryptionkey@0"
I20260806 15:58:50.878711 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:50.899714 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:50.903138 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:50.903352 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:50.910815 17145 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:36187
I20260806 15:58:50.911098 17251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:36187 every 8 connection(s)
I20260806 15:58:50.911788 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:50.914623 17252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:50.918474 17252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:50.918900 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17145
I20260806 15:58:50.919030 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:50.919328 14204 external_mini_cluster.cc:1428] Setting key eeddb028ebba75afc53a21827af5c24c
WARNING: no policy specified for kudu/127.13.223.61@KRBTEST.COM; defaulting to no policy
Aug 06 15:58:50 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.61@KRBTEST.COM" created.
I20260806 15:58:50.937230 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:50.921019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:36187 (local address 127.0.0.1:51922)
0806 15:58:50.921768 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:50.921823 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:50.923188 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:50.923916 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:50.923921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:50.924396 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:50.924708 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:50.924722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:50.926798 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:50.926801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:50.927496 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:50.927504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:50.928564 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:50.928569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:50.928748 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:50.929425 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:50.929440 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:50.930911 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:50.933678 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:50.933688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:50.933699 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:50.934025 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:50.934414 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:50.934420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:50.934424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:50.934577 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:50.935240 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:50.935247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:50.935400 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 15:58:50.935651 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 15:58:50.936068 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":188,"thread_start_us":153,"threads_started":1}
I20260806 15:58:50.937230 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:50.921018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.13.223.62:36187)
0806 15:58:50.921430 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:50.921452 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:50.921996 (+ 544us) server_negotiation.cc:407] Connection header received
0806 15:58:50.923531 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:50.923536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:50.923726 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:50.923918 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:50.924957 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:50.926649 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:50.927642 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:50.928336 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:50.928644 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:50.931074 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:50.931080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:50.931091 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:50.931115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:50.933559 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:50.934170 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:50.934175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:50.934176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:50.934298 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:50.934693 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:50.934696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:50.934697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:50.935101 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:50.935270 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:50.935740 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 15:58:50.936029 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":262,"spinlock_wait_cycles":4352,"thread_start_us":128,"threads_started":1}
I20260806 15:58:50.939706 17252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:50.940434 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:50.940181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60686)
0806 15:58:50.940298 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:50.940376 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 15:58:50.940649 17169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
I20260806 15:58:50.941473 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:50.941254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60692)
0806 15:58:50.941370 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:50.941422 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 15:58:50.941602 17252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:37509: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab.
Entry for principal kudu/127.13.223.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab.
I20260806 15:58:50.951910 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:37509
--webserver_interface=127.13.223.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab
--principal=kudu/127.13.223.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:36187,127.13.223.61:37509,127.13.223.60:44591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:58:50.990841 17252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } attempt: 1
I20260806 15:58:50.991758 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:50.991449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60698)
0806 15:58:50.991602 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:50.991678 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 15:58:50.992569 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:50.992304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60702)
0806 15:58:50.992412 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:50.992465 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 15:58:50.992730 17252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:37509: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
I20260806 15:58:51.060887 17252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } attempt: 2
I20260806 15:58:51.061879 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.061469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60718)
0806 15:58:51.061705 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.061779 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 15:58:51.062767 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.062527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60728)
0806 15:58:51.062662 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.062709 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 15:58:51.062906 17252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:37509: Network error: Client connection negotiation failed: client connection to 127.13.223.61:37509: connect: Connection refused (error 111)
W20260806 15:58:51.067528 17260 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 15:58:51.067950 17260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:51.068024 17260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:51.068120 17260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:51.072804 17260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:51.072889 17260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:51.072943 17260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:51.073026 17260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:51.073073 17260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:51.079169 17260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:36187,127.13.223.61:37509,127.13.223.60:44591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab
--principal=kudu/127.13.223.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:51.080396 17260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:51.081674 17260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:51.091034 17267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:51.091030 17265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:51.091025 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:51.091351 17260 server_base.cc:1034] running on GCE node
I20260806 15:58:51.092054 17260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:51.093298 17260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:51.094556 17260 hybrid_clock.cc:648] HybridClock initialized: now 1786031931094527 us; error 42 us; skew 500 ppm
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:51.097491 17260 init.cc:377] Logged in from keytab as kudu/127.13.223.61@KRBTEST.COM (short username kudu)
I20260806 15:58:51.098433 17260 webserver.cc:460] Webserver started at http://127.13.223.61:41005/ using document root <none> and password file <none>
I20260806 15:58:51.098994 17260 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:51.099090 17260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:51.099328 17260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:51.101454 17260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data/instance:
uuid: "d240bf867905486586012a0b68dd44cd"
format_stamp: "Formatted at 2026-08-06 15:58:51 on dist-test-slave-rwrg"
server_key: "f1697d91baec3493a05b51f865144e90"
server_key_iv: "5169260e250255c2621af8b128419db4"
server_key_version: "encryptionkey@0"
I20260806 15:58:51.102006 17260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal/instance:
uuid: "d240bf867905486586012a0b68dd44cd"
format_stamp: "Formatted at 2026-08-06 15:58:51 on dist-test-slave-rwrg"
server_key: "f1697d91baec3493a05b51f865144e90"
server_key_iv: "5169260e250255c2621af8b128419db4"
server_key_version: "encryptionkey@0"
I20260806 15:58:51.105744 17260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:58:51.108043 17282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:51.109325 17260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:58:51.109467 17260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal
uuid: "d240bf867905486586012a0b68dd44cd"
format_stamp: "Formatted at 2026-08-06 15:58:51 on dist-test-slave-rwrg"
server_key: "f1697d91baec3493a05b51f865144e90"
server_key_iv: "5169260e250255c2621af8b128419db4"
server_key_version: "encryptionkey@0"
I20260806 15:58:51.109576 17260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:51.122980 17260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:51.126072 17260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:51.126284 17260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:51.133476 17260 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.61:37509
I20260806 15:58:51.133781 17356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.61:37509 every 8 connection(s)
I20260806 15:58:51.134471 17260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
I20260806 15:58:51.137236 17357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:51.138952 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17260
I20260806 15:58:51.139089 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-1/wal/instance
I20260806 15:58:51.139386 14204 external_mini_cluster.cc:1428] Setting key db4357bb90c61eb98a717bd24f3e64ba
I20260806 15:58:51.142057 17357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.60@KRBTEST.COM" created.
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:51.154074 17252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } attempt: 3
I20260806 15:58:51.158644 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.144079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.13.223.62:36187)
0806 15:58:51.144247 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.144252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.145196 (+ 944us) server_negotiation.cc:407] Connection header received
0806 15:58:51.146156 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.146161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.146222 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.146337 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.147475 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.148397 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.149836 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.150095 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.150187 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.153105 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.153109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.153111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.153138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.155870 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.156581 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.156587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.156592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.156687 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.157126 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.157131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.157135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.157398 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.157509 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.158317 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.158451 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 15:58:51.159715 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.144435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:36187 (local address 127.0.0.1:51928)
0806 15:58:51.145005 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.145059 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.145924 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.146493 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.146500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.146829 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.147313 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.147326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.148634 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.148638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.149703 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.149714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.150269 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.150273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.150424 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.151172 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.151190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.152882 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.156052 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.156059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.156080 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.156447 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.156801 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.156804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.156806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.156990 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.157580 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.157590 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.157946 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.158172 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.158527 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":155,"threads_started":1}
I20260806 15:58:51.161453 17357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
I20260806 15:58:51.167743 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.154599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60734)
0806 15:58:51.154860 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.154876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.154957 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.155724 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.155730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.155965 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.156136 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.156147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.157499 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.157503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.158074 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.158083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.159205 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.159209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.159334 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.160294 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.160307 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.163674 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.165977 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.165983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.165987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.166298 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.166664 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.166674 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.166676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.166806 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.167265 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.167271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.167394 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.167490 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.167601 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260806 15:58:51.168567 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.154723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.13.223.61:37509)
0806 15:58:51.155169 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.155185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.155214 (+ 29us) server_negotiation.cc:407] Connection header received
0806 15:58:51.155265 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.155273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.155523 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.155720 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.156276 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.157381 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.158213 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.159070 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.159204 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.163804 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.163815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.163829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.163854 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.165841 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.166425 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.166431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.166433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.166515 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.166902 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.166905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.166906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.167129 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.167277 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.167607 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.167760 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab.
Entry for principal kudu/127.13.223.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab.
I20260806 15:58:51.170655 17252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:51.171519 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.171188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42040)
0806 15:58:51.171374 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.171447 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 15:58:51.171581 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:44591
--webserver_interface=127.13.223.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab
--principal=kudu/127.13.223.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:36187,127.13.223.61:37509,127.13.223.60:44591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:58:51.171780 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.161973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60740)
0806 15:58:51.162160 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.162181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.162242 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.162678 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.162680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.162846 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.162920 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.162925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.163819 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.163822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.164672 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.164686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.165378 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.165381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.165484 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.165913 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.165930 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.167495 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.170125 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.170129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.170130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.170380 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.170676 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.170682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.170687 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.170774 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.171242 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.171245 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.171338 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.171465 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.171620 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 15:58:51.172008 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.162172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.13.223.61:37509)
0806 15:58:51.162461 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.162471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.162481 (+ 10us) server_negotiation.cc:407] Connection header received
0806 15:58:51.162506 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.162508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.162537 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.162672 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.163031 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.163693 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.164812 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.165240 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.165356 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.167682 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.167686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.167688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.167709 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.170012 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.170491 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.170497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.170501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.170585 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.170888 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.170892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.170893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.171120 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.171242 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.171514 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.171814 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260806 15:58:51.172326 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.171995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42044)
0806 15:58:51.172194 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.172250 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260806 15:58:51.172447 17252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:44591: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
I20260806 15:58:51.172691 17357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:51.173354 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.173099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42046)
0806 15:58:51.173217 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.173301 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 15:58:51.173550 17285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
I20260806 15:58:51.174381 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.174146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42058)
0806 15:58:51.174258 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.174308 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 15:58:51.174520 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:44591: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
I20260806 15:58:51.203685 17252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } attempt: 1
I20260806 15:58:51.204680 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.204286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42064)
0806 15:58:51.204455 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.204580 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 15:58:51.205504 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.205273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42078)
0806 15:58:51.205390 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.205447 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 15:58:51.205682 17252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:44591: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
I20260806 15:58:51.223786 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } attempt: 1
I20260806 15:58:51.224647 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.224343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42088)
0806 15:58:51.224480 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.224571 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 15:58:51.225387 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.225177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42100)
0806 15:58:51.225281 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.225328 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 15:58:51.225541 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:44591: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
I20260806 15:58:51.280898 17252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } attempt: 2
I20260806 15:58:51.281805 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.281463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42110)
0806 15:58:51.281627 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.281702 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 15:58:51.282791 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.282518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42122)
0806 15:58:51.282647 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.282699 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 15:58:51.282990 17252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:44591: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
W20260806 15:58:51.287688 17376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 15:58:51.287930 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:51.287952 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:51.287993 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:51.291914 17376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:51.291963 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:51.291981 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:51.291997 17376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:51.292012 17376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:51.293707 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } attempt: 2
I20260806 15:58:51.294498 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.294193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42130)
0806 15:58:51.294361 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.294427 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 15:58:51.295282 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.295052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42140)
0806 15:58:51.295167 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.295224 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 15:58:51.295441 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:44591: Network error: Client connection negotiation failed: client connection to 127.13.223.60:44591: connect: Connection refused (error 111)
I20260806 15:58:51.297308 17376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:36187,127.13.223.61:37509,127.13.223.60:44591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab
--principal=kudu/127.13.223.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:44591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:51.298311 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:51.299397 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:51.308172 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:51.308322 17391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:51.308187 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:51.308715 17376 server_base.cc:1034] running on GCE node
I20260806 15:58:51.309232 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:51.310281 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:51.311463 17376 hybrid_clock.cc:648] HybridClock initialized: now 1786031931311426 us; error 49 us; skew 500 ppm
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:51.314519 17376 init.cc:377] Logged in from keytab as kudu/127.13.223.60@KRBTEST.COM (short username kudu)
I20260806 15:58:51.315414 17376 webserver.cc:460] Webserver started at http://127.13.223.60:39007/ using document root <none> and password file <none>
I20260806 15:58:51.315972 17376 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:51.316036 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:51.316294 17376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:51.318397 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data/instance:
uuid: "2db487c7847f4b4381e12ac7380ed179"
format_stamp: "Formatted at 2026-08-06 15:58:51 on dist-test-slave-rwrg"
server_key: "668bfa649ad9bda6f3ac8e48c86949be"
server_key_iv: "9a475a663d49461bb8d219df1a6d5d35"
server_key_version: "encryptionkey@0"
I20260806 15:58:51.318961 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal/instance:
uuid: "2db487c7847f4b4381e12ac7380ed179"
format_stamp: "Formatted at 2026-08-06 15:58:51 on dist-test-slave-rwrg"
server_key: "668bfa649ad9bda6f3ac8e48c86949be"
server_key_iv: "9a475a663d49461bb8d219df1a6d5d35"
server_key_version: "encryptionkey@0"
I20260806 15:58:51.322784 17376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 15:58:51.325268 17401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:51.326591 17376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 15:58:51.326730 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal
uuid: "2db487c7847f4b4381e12ac7380ed179"
format_stamp: "Formatted at 2026-08-06 15:58:51 on dist-test-slave-rwrg"
server_key: "668bfa649ad9bda6f3ac8e48c86949be"
server_key_iv: "9a475a663d49461bb8d219df1a6d5d35"
server_key_version: "encryptionkey@0"
I20260806 15:58:51.326841 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:51.340736 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:51.344130 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:51.344340 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:51.351890 17376 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.60:44591
I20260806 15:58:51.352216 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.60:44591 every 8 connection(s)
I20260806 15:58:51.352929 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
I20260806 15:58:51.355690 17495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:51.358944 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17376
I20260806 15:58:51.359086 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786031632208048-14204-0/minicluster-data/master-2/wal/instance
I20260806 15:58:51.359404 14204 external_mini_cluster.cc:1428] Setting key 4ca1d04eb0f3978cd986a462e2436394
I20260806 15:58:51.359793 17495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:51.373234 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.361866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.13.223.62:36187)
0806 15:58:51.362011 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.362016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.362245 (+ 229us) server_negotiation.cc:407] Connection header received
0806 15:58:51.362622 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.362629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.362683 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.362771 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.363855 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.364707 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.365961 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.366175 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.366251 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.369628 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.369632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.369634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.369658 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.371057 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.371581 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.371586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.371588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.371693 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.372066 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.372072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.372074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.372281 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.372401 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.372894 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.373051 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 15:58:51.373960 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.360782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.13.223.60:44591)
0806 15:58:51.361227 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.361243 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.361279 (+ 36us) server_negotiation.cc:407] Connection header received
0806 15:58:51.362114 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.362134 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.362425 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.362691 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.363668 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.364977 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.365895 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.366372 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.366506 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.369048 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.369056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.369071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.369104 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.370999 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.371458 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.371463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.371464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.371567 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.371853 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.371856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.371858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.372232 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.372518 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.372718 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.373036 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":301,"spinlock_wait_cycles":2432,"thread_start_us":128,"threads_started":1}
I20260806 15:58:51.373960 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.361784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:36187 (local address 127.0.0.1:51932)
0806 15:58:51.362084 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.362141 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.362427 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.362849 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.362856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.363355 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.363711 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.363725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.364855 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.364858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.365802 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.365812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.366333 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.366338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.366509 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.367360 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.367378 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.369457 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.371190 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.371203 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.371214 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.371464 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.371810 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.371813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.371815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.371950 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.372402 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.372409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.372639 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.372804 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.373121 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":148,"spinlock_wait_cycles":4736,"thread_start_us":85,"threads_started":1}
I20260806 15:58:51.375939 17495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:51.382210 17252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } attempt: 3
I20260806 15:58:51.385650 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.376478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:37509 (local address 127.0.0.1:60752)
0806 15:58:51.376666 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.376679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.376778 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.377065 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.377068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.377254 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.377362 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.377368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.378331 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.378335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.378987 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.378997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.379617 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.379621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.379722 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.380214 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.380233 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.381594 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.383921 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.383925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.383927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.384196 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.384554 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.384558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.384560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.384657 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.385136 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.385139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.385240 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.385348 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.385478 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 15:58:51.385751 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.376709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.13.223.61:37509)
0806 15:58:51.376814 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.376818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.376835 (+ 17us) server_negotiation.cc:407] Connection header received
0806 15:58:51.376884 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.376888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.376938 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.377017 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.377492 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.378221 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.379118 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.379483 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.379591 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.381873 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.381876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.381878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.381901 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.383547 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.384317 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.384322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.384325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.384431 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.384771 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.384775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.384777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.384999 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.385131 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.385446 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.385576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 15:58:51.386444 17495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:51.386593 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } attempt: 3
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
I20260806 15:58:51.388633 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.376569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.13.223.62:36187)
0806 15:58:51.376682 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.376686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.376908 (+ 222us) server_negotiation.cc:407] Connection header received
0806 15:58:51.377071 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.377075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.377124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.377218 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.377993 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.378613 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.379589 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.380008 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.380150 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.383353 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.383356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.383359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.383383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.385306 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.386351 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.386375 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.386379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.386490 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.387395 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.387399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.387401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.387616 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.387859 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.388237 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.388418 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 15:58:51.392936 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.382738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42152)
0806 15:58:51.383109 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.383127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.383217 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.383523 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.383526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.383712 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.383799 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.383805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.384819 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.384823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.385618 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.385629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.386312 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.386316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.386422 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.387160 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.387177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.388812 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.390945 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.390949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.390951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.391213 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.391788 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.391791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.391793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.391886 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.392348 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.392352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.392477 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.392628 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.392774 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260806 15:58:51.393630 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.382957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.13.223.60:44591)
0806 15:58:51.383111 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.383117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.383174 (+ 57us) server_negotiation.cc:407] Connection header received
0806 15:58:51.383336 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.383339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.383384 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.383468 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.383921 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.384723 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.385771 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.386158 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.386657 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.388964 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.388967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.388971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.388999 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.390820 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.391328 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.391330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.391332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.391415 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.392025 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.392028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.392029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.392216 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.392336 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.393301 (+ 965us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.393430 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:58:51 dist-test-slave-rwrg krb5kdc[17125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
I20260806 15:58:51.397907 17252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24: Bootstrap starting.
I20260806 15:58:51.399056 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.386948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42160)
0806 15:58:51.387397 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.387412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.387477 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.388974 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.388977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.389534 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.389625 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.389632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.390700 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.390703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.391675 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.391689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.392179 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.392182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.392264 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.392746 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.392760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.394902 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.397312 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.397315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.397318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.397577 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.397873 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.397876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.397877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.397984 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.398491 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.398493 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.398603 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.398766 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.398898 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":46}
I20260806 15:58:51.399739 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.387066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.13.223.60:44591)
0806 15:58:51.388667 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.388676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.388690 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:51.388723 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.388726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.388767 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.388855 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.389748 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.390584 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.391782 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.392063 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.392178 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.395067 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.395071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.395073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.395098 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.397188 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.397688 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.397692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.397694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.397778 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.398093 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.398096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.398098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.398355 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.398455 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.399392 (+ 937us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.399567 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1530,"thread_start_us":148,"threads_started":1}
I20260806 15:58:51.401206 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.390274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.13.223.61:37509)
0806 15:58:51.390377 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.390381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.390672 (+ 291us) server_negotiation.cc:407] Connection header received
0806 15:58:51.390713 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.390716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.390756 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.390827 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.391738 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.392473 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.393661 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.394003 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.394120 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.397196 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.397200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.397202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.397228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.399056 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.399589 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.399596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.399598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.399692 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.400033 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.400037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.400038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.400252 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.400450 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.400872 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.401055 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 15:58:51.401207 17252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:51.401589 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.386983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:44591 (local address 127.0.0.1:42172)
0806 15:58:51.387104 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:51.387120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:51.387196 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:51.387635 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:51.387638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:51.387813 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:51.387914 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.387921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.389511 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.389515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.390408 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:51.390421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.391201 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.391204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:51.391303 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.392591 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:51.392612 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:51.396857 (+ 4245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:51.399762 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.399768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.399772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.400034 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.400580 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:51.400583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:51.400585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:51.400674 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:51.401127 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:51.401131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:51.401218 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 15:58:51.401304 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 15:58:51.401435 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 15:58:51.402204 17252 log.cc:824] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:51.402828 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.386977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.13.223.60:44591)
0806 15:58:51.387397 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.387411 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.387426 (+ 15us) server_negotiation.cc:407] Connection header received
0806 15:58:51.387462 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.387465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.387498 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.387617 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.388049 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.389394 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.390570 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.391064 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.391504 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.397076 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.397083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.397089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.397121 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.399641 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.400357 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.400359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.400361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.400442 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.400796 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.400801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.400806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.401000 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.402465 (+ 1465us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.402550 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.402675 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":341,"thread_start_us":133,"threads_started":1}
I20260806 15:58:51.403002 17495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: Bootstrap starting.
I20260806 15:58:51.404582 17252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24: No bootstrap required, opened a new log
I20260806 15:58:51.406040 17357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: Bootstrap starting.
I20260806 15:58:51.406406 17495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:51.407470 17495 log.cc:824] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:51.408363 17357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:51.409049 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:51.409310 17357 log.cc:824] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:51.409212 17252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.409480 17252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be12b4b51ab4939a30f05b1cac93e24, State: Initialized, Role: FOLLOWER
I20260806 15:58:51.409834 17252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.409941 17495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: No bootstrap required, opened a new log
I20260806 15:58:51.410490 17510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.410508 17210 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7be12b4b51ab4939a30f05b1cac93e24"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51936
I20260806 15:58:51.410815 17510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.411307 17252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:51.411412 17210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 15:58:51.411572 17210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:51.412879 17210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.413904 17210 leader_election.cc:288] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [CANDIDATE]: Term 1 election: Requested vote from peers d240bf867905486586012a0b68dd44cd (127.13.223.61:37509), 2db487c7847f4b4381e12ac7380ed179 (127.13.223.60:44591)
I20260806 15:58:51.415354 17495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.415635 17495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db487c7847f4b4381e12ac7380ed179, State: Initialized, Role: FOLLOWER
I20260806 15:58:51.416002 17495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.416646 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.416836 17357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: No bootstrap required, opened a new log
I20260806 15:58:51.416985 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.417550 17495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:51.417584 17447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7be12b4b51ab4939a30f05b1cac93e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2db487c7847f4b4381e12ac7380ed179"
I20260806 15:58:51.417793 17447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:51.419250 17447 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7be12b4b51ab4939a30f05b1cac93e24 in term 1.
I20260806 15:58:51.418867 17145 master_runner.cc:186] Error getting master registration for 127.13.223.61:37509: OK, instance_id { permanent_uuid: "d240bf867905486586012a0b68dd44cd" instance_seqno: 1786031931131485 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 15:58:51.422761 17145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 15:58:51.423156 17172 leader_election.cc:302] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db487c7847f4b4381e12ac7380ed179, 7be12b4b51ab4939a30f05b1cac93e24; no voters:
I20260806 15:58:51.423072 17357 raft_consensus.cc:348] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.423307 17357 raft_consensus.cc:729] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d240bf867905486586012a0b68dd44cd, State: Initialized, Role: FOLLOWER
I20260806 15:58:51.423437 17510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:51.423671 17510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [term 1 LEADER]: Becoming Leader. State: Replica: 7be12b4b51ab4939a30f05b1cac93e24, State: Running, Role: LEADER
I20260806 15:58:51.423671 17357 consensus_queue.cc:260] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.424018 17510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } }
I20260806 15:58:51.425187 17357 sys_catalog.cc:564] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:51.425935 17510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be12b4b51ab4939a30f05b1cac93e24. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.426075 17510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:51.426512 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7be12b4b51ab4939a30f05b1cac93e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d240bf867905486586012a0b68dd44cd"
I20260806 15:58:51.426749 17320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:51.428308 17320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7be12b4b51ab4939a30f05b1cac93e24 in term 1.
W20260806 15:58:51.430836 17528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:51.430912 17528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:51.431169 17559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.431303 17559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.432160 17376 master_runner.cc:186] Error getting master registration for 127.13.223.61:37509: OK, instance_id { permanent_uuid: "d240bf867905486586012a0b68dd44cd" instance_seqno: 1786031931131485 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 15:58:51.436296 17376 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 15:58:51.423086 17554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:51.436555 17554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:51.438423 17574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:51.439491 17574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:51.449123 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.425268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.13.223.61:37509)
0806 15:58:51.425371 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.425376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.432775 (+ 7399us) server_negotiation.cc:407] Connection header received
0806 15:58:51.434949 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.434952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.434997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.435047 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.437050 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.437732 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.442138 (+ 4406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.442548 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.442679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.444670 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.444676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.444680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.444713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.446747 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.447400 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.447410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.447413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.447516 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.447890 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.447895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.447897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.448186 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.448327 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.448757 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.448896 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 15:58:51.451892 17586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:51.451982 17586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:51.452152 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.426750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.13.223.60:44591)
0806 15:58:51.426874 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.426878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.426894 (+ 16us) server_negotiation.cc:407] Connection header received
0806 15:58:51.427363 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.427368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.427425 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.427479 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.430905 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.431646 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.436403 (+ 4757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.436866 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.436942 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.443466 (+ 6524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.443474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.443478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.443514 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.445962 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.449664 (+ 3702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.449670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.449673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.449775 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.451071 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.451078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.451083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.451378 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.451476 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.451805 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.451947 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 15:58:51.452456 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:51.424589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.13.223.62:36187)
0806 15:58:51.424704 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.424709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.427279 (+ 2570us) server_negotiation.cc:407] Connection header received
0806 15:58:51.428857 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.428861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.428904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.428953 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.432877 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.433672 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.437861 (+ 4189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:51.438163 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:51.438268 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:51.446189 (+ 7921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:51.446193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:51.446196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:51.446227 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:51.448198 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.449146 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.449150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.449152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.449267 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:51.450317 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:51.450320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:51.450322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:51.450578 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:51.450722 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:51.452120 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0806 15:58:51.452256 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 15:58:51.457489 17447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [term 1 FOLLOWER]: Refusing update from remote peer 7be12b4b51ab4939a30f05b1cac93e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:51.457803 17320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [term 1 FOLLOWER]: Refusing update from remote peer 7be12b4b51ab4939a30f05b1cac93e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:51.458451 17510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:51.458940 17510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:51.463526 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be12b4b51ab4939a30f05b1cac93e24. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.463660 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.464325 17559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be12b4b51ab4939a30f05b1cac93e24. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.464461 17559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.466683 17510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.466811 17510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:51.467028 17559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.467160 17559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.467453 17588 mvcc.cc:205] Tried to move back new op lower bound from 7315586791242510336 to 7315586791116492800. Current Snapshot: MvccSnapshot[applied={T|T < 7315586791242510336}]
I20260806 15:58:51.469405 17595 mvcc.cc:205] Tried to move back new op lower bound from 7315586791242510336 to 7315586791116492800. Current Snapshot: MvccSnapshot[applied={T|T < 7315586791242510336}]
I20260806 15:58:51.471254 17574 catalog_manager.cc:1347] Generated new cluster ID: 4a48715d793448638d80fd1bbad27ec8
I20260806 15:58:51.471341 17574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:51.478601 17558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.478776 17558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:51.480000 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7be12b4b51ab4939a30f05b1cac93e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7be12b4b51ab4939a30f05b1cac93e24" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 36187 } } peers { permanent_uuid: "d240bf867905486586012a0b68dd44cd" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 37509 } } peers { permanent_uuid: "2db487c7847f4b4381e12ac7380ed179" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 44591 } } }
I20260806 15:58:51.480100 17592 mvcc.cc:205] Tried to move back new op lower bound from 7315586791242510336 to 7315586791116492800. Current Snapshot: MvccSnapshot[applied={T|T < 7315586791242510336}]
I20260806 15:58:51.480144 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:51.515192 17574 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:51.516808 17574 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:51.532431 17574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7be12b4b51ab4939a30f05b1cac93e24: Generated new TSK 0
I20260806 15:58:51.533435 17574 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 15:58:51.563059 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.559840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.13.223.62:36187)
0806 15:58:51.559987 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.559994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.561960 (+ 1966us) server_negotiation.cc:407] Connection header received
0806 15:58:51.562168 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.562172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.562224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.562278 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.562906 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51946: BlockingRecv error: recv got EOF from 127.0.0.1:51946 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 15:58:51.565323 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.561143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.13.223.61:37509)
0806 15:58:51.561257 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.561261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.563772 (+ 2511us) server_negotiation.cc:407] Connection header received
0806 15:58:51.564616 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.564619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.564671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.564721 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.565205 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60776: BlockingRecv error: recv got EOF from 127.0.0.1:60776 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 15:58:51.565836 17504 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.560347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.13.223.60:44591)
0806 15:58:51.560478 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.560482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.563446 (+ 2964us) server_negotiation.cc:407] Connection header received
0806 15:58:51.564043 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.564046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.564100 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.564150 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.565722 (+ 1572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42198: BlockingRecv error: recv got EOF from 127.0.0.1:42198 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 15:58:51.570295 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.568716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.13.223.61:37509)
0806 15:58:51.568816 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.568820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.569355 (+ 535us) server_negotiation.cc:407] Connection header received
0806 15:58:51.569566 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.569569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.569652 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.569704 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.570200 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60786: BlockingRecv error: recv got EOF from 127.0.0.1:60786 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 15:58:51.575232 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.573835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.13.223.61:37509)
0806 15:58:51.573934 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.573938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.574400 (+ 462us) server_negotiation.cc:407] Connection header received
0806 15:58:51.574588 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.574591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.574636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.574686 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.575151 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60790: BlockingRecv error: recv got EOF from 127.0.0.1:60790 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260806 15:58:51.580847 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.579321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.13.223.61:37509)
0806 15:58:51.579417 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.579421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.579983 (+ 562us) server_negotiation.cc:407] Connection header received
0806 15:58:51.580161 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.580164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.580210 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.580258 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.580755 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60800: BlockingRecv error: recv got EOF from 127.0.0.1:60800 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260806 15:58:51.588320 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.586885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.13.223.61:37509)
0806 15:58:51.587014 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.587017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.587455 (+ 438us) server_negotiation.cc:407] Connection header received
0806 15:58:51.587640 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.587643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.587688 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.587734 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.588229 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60812: BlockingRecv error: recv got EOF from 127.0.0.1:60812 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 15:58:51.600070 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.598560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.13.223.61:37509)
0806 15:58:51.598675 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.598679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.599176 (+ 497us) server_negotiation.cc:407] Connection header received
0806 15:58:51.599362 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.599366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.599412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.599460 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.599974 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60820: BlockingRecv error: recv got EOF from 127.0.0.1:60820 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 15:58:51.620678 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.618694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.13.223.61:37509)
0806 15:58:51.618838 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.618842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.619512 (+ 670us) server_negotiation.cc:407] Connection header received
0806 15:58:51.619770 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.619774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.619831 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.619889 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.620534 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60830: BlockingRecv error: recv got EOF from 127.0.0.1:60830 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 15:58:51.657538 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.655607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.13.223.61:37509)
0806 15:58:51.655765 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.655770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.656433 (+ 663us) server_negotiation.cc:407] Connection header received
0806 15:58:51.656721 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.656725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.656782 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.656836 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.657409 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60836: BlockingRecv error: recv got EOF from 127.0.0.1:60836 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 15:58:51.725556 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.724220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.13.223.61:37509)
0806 15:58:51.724403 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.724407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.724638 (+ 231us) server_negotiation.cc:407] Connection header received
0806 15:58:51.724708 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.724711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.724764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.724853 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.725415 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60848: BlockingRecv error: recv got EOF from 127.0.0.1:60848 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 15:58:51.857044 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:51.855848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.13.223.61:37509)
0806 15:58:51.856020 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:51.856024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:51.856044 (+ 20us) server_negotiation.cc:407] Connection header received
0806 15:58:51.856240 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:51.856244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:51.856288 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:51.856376 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:51.856894 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60860: BlockingRecv error: recv got EOF from 127.0.0.1:60860 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 15:58:52.116585 17371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:52.115280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.13.223.61:37509)
0806 15:58:52.115437 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:52.115442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:52.115548 (+ 106us) server_negotiation.cc:407] Connection header received
0806 15:58:52.115762 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:52.115765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:52.115824 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:52.115936 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:52.116435 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60874: BlockingRecv error: recv got EOF from 127.0.0.1:60874 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 15:58:52.437970 17554 catalog_manager.cc:1259] Loaded cluster ID: 4a48715d793448638d80fd1bbad27ec8
I20260806 15:58:52.438107 17554 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: loading cluster ID for follower catalog manager: success
I20260806 15:58:52.439829 17554 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: acquiring CA information for follower catalog manager: success
I20260806 15:58:52.441064 17554 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2db487c7847f4b4381e12ac7380ed179: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 15:58:52.453234 17586 catalog_manager.cc:1259] Loaded cluster ID: 4a48715d793448638d80fd1bbad27ec8
I20260806 15:58:52.453307 17586 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: loading cluster ID for follower catalog manager: success
I20260806 15:58:52.455083 17586 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: acquiring CA information for follower catalog manager: success
I20260806 15:58:52.456321 17586 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d240bf867905486586012a0b68dd44cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 15:58:52.635006 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:52.630960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.13.223.61:37509)
0806 15:58:52.631339 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:52.631345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:52.631365 (+ 20us) server_negotiation.cc:407] Connection header received
0806 15:58:52.631439 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:52.631441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:52.631565 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:52.631692 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:52.631999 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.632546 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.633468 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.633697 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.633811 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:52.634141 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:52.634591 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:52.634729 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:52.634771 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 15:58:52.634824 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":145,"threads_started":1}
I20260806 15:58:52.642309 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:52.638492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.13.223.60:44591)
0806 15:58:52.638804 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:52.638810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:52.638824 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:52.638909 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:52.638911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:52.639019 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:52.639143 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:52.639377 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.639892 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.640825 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.641057 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.641168 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:52.641453 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:52.641867 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:52.642000 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:52.642074 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 15:58:52.642129 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260806 15:58:52.649708 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:52.645707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.13.223.61:37509)
0806 15:58:52.645904 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:52.645908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:52.646128 (+ 220us) server_negotiation.cc:407] Connection header received
0806 15:58:52.646271 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:52.646274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:52.646355 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:52.646423 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:52.646759 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.647252 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.648482 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.648831 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.648948 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:52.649213 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:52.649289 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:52.649373 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:52.649493 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 15:58:52.649550 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 15:58:52.650182 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:52.645522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.13.223.60:44591)
0806 15:58:52.645672 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:52.645676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:52.646047 (+ 371us) server_negotiation.cc:407] Connection header received
0806 15:58:52.646212 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:52.646215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:52.646266 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:52.646343 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:52.646598 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.647350 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.649073 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.649388 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.649483 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:52.649728 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:52.649801 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:52.649871 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:52.650010 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 15:58:52.650065 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 15:58:52.650305 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:52.645358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.13.223.62:36187)
0806 15:58:52.645816 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:52.645820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:52.645832 (+ 12us) server_negotiation.cc:407] Connection header received
0806 15:58:52.645903 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:52.645905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:52.646026 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:52.646440 (+ 414us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:52.646539 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.647351 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.648549 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:52.648894 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:52.649293 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:52.649425 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:52.649767 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:52.650041 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:52.650074 (+ 33us) server_negotiation.cc:299] Negotiation successful
0806 15:58:52.650154 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":91,"threads_started":1}
I20260806 15:58:52.653690 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17145
I20260806 15:58:52.671666 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17260
I20260806 15:58:52.682670 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17376
2026-08-06T15:58:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3359 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 15:58:52 dist-test-slave-rwrg krb5kdc[17753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 15:58:52 dist-test-slave-rwrg krb5kdc[17753](info): set up 2 sockets
Aug 06 15:58:52 dist-test-slave-rwrg krb5kdc[17753](info): commencing operation
krb5kdc: starting...
W20260806 15:58:53.938803 14204 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/test-user.keytab.
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T15:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T15:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
Entry for principal kudu/127.13.223.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab.
I20260806 15:58:54.069446 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:38441
--webserver_interface=127.13.223.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:38441,127.13.223.61:45589,127.13.223.60:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 15:58:54.182782 17776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 15:58:54.183040 17776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:54.183063 17776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:54.183099 17776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 15:58:54.187011 17776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:54.187093 17776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:54.187111 17776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:54.187127 17776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:54.187142 17776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:54.192237 17776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:38441,127.13.223.61:45589,127.13.223.60:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.62.keytab
--principal=kudu/127.13.223.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.62:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:54.193306 17776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:54.194253 17776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:54.202793 17792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:54.202773 17787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:54.202876 17776 server_base.cc:1034] running on GCE node
W20260806 15:58:54.202772 17788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:54.203743 17776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:54.204877 17776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:54.206068 17776 hybrid_clock.cc:648] HybridClock initialized: now 1786031934206050 us; error 31 us; skew 500 ppm
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:54.208786 17776 init.cc:377] Logged in from keytab as kudu/127.13.223.62@KRBTEST.COM (short username kudu)
I20260806 15:58:54.209717 17776 webserver.cc:460] Webserver started at http://127.13.223.62:41063/ using document root <none> and password file <none>
I20260806 15:58:54.210264 17776 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:54.210361 17776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:54.210603 17776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:54.212669 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data/instance:
uuid: "770c099adf844e7896891ac13e4a6e17"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "a77857f9283f5fb36bb4cfd711955ae5"
server_key_iv: "79aabf095f8c8099945443d5c1ff1da0"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.213229 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal/instance:
uuid: "770c099adf844e7896891ac13e4a6e17"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "a77857f9283f5fb36bb4cfd711955ae5"
server_key_iv: "79aabf095f8c8099945443d5c1ff1da0"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.217015 17776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 15:58:54.219316 17799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:54.220615 17776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:54.220767 17776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal
uuid: "770c099adf844e7896891ac13e4a6e17"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "a77857f9283f5fb36bb4cfd711955ae5"
server_key_iv: "79aabf095f8c8099945443d5c1ff1da0"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.220876 17776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:54.233807 17776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:54.236479 17776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:54.236704 17776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:54.243558 17776 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.62:38441
I20260806 15:58:54.243852 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.62:38441 every 8 connection(s)
I20260806 15:58:54.244594 17776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/data/info.pb
I20260806 15:58:54.246560 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17776
I20260806 15:58:54.246697 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-0/wal/instance
I20260806 15:58:54.247015 14204 external_mini_cluster.cc:1428] Setting key 8d527dd302157599419ee5fd3bbf70cf
I20260806 15:58:54.247566 17884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:54.251965 17884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.61@KRBTEST.COM" created.
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:54.267504 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.254265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.13.223.62:38441)
0806 15:58:54.254843 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.254855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.254904 (+ 49us) server_negotiation.cc:407] Connection header received
0806 15:58:54.255969 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.255977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.256188 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.256346 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.257213 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.258275 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.259497 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.259884 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.260055 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.262625 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.262632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.262652 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.262672 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.264324 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.264858 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.264863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.264865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.264967 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.265295 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.265297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.265299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.265643 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.265780 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.266155 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.266433 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":463,"thread_start_us":173,"threads_started":1}
I20260806 15:58:54.267504 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.254264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38441 (local address 127.0.0.1:38330)
0806 15:58:54.254747 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.254800 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.255629 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.256373 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.256378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.256793 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.257044 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.257058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.258454 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.258460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.259365 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.259375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.260042 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.260046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.260156 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.260814 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.260830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.262474 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.264453 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.264464 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.264474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.264749 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.265079 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.265083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.265085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.265198 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.265762 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.265768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.265906 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.266073 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.266483 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":142,"thread_start_us":131,"threads_started":1}
I20260806 15:58:54.269896 17884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:54.270658 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.270339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38750)
0806 15:58:54.270454 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.270555 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 15:58:54.270895 17800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
I20260806 15:58:54.271751 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.271501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38758)
0806 15:58:54.271611 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.271654 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 15:58:54.271919 17884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:45589: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab.
Entry for principal kudu/127.13.223.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab.
I20260806 15:58:54.274902 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:45589
--webserver_interface=127.13.223.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab
--principal=kudu/127.13.223.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:38441,127.13.223.61:45589,127.13.223.60:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 15:58:54.321156 17884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } attempt: 1
I20260806 15:58:54.322157 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.321794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38764)
0806 15:58:54.321998 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.322074 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 15:58:54.322981 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.322753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38776)
0806 15:58:54.322868 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.322925 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 15:58:54.323118 17884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:45589: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
W20260806 15:58:54.387284 17892 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 15:58:54.387625 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:54.387697 17892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:54.387780 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 15:58:54.391321 17884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } attempt: 2
W20260806 15:58:54.391875 17892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:54.391947 17892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:54.392026 17892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:54.392087 17892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:54.392117 17892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:54.392282 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.391893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38778)
0806 15:58:54.392077 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.392154 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 15:58:54.393100 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.392844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38794)
0806 15:58:54.392962 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.393037 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 15:58:54.393239 17884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.61:45589: Network error: Client connection negotiation failed: client connection to 127.13.223.61:45589: connect: Connection refused (error 111)
I20260806 15:58:54.397635 17892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:38441,127.13.223.61:45589,127.13.223.60:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.61.keytab
--principal=kudu/127.13.223.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.61:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:54.398674 17892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:54.399727 17892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 15:58:54.408360 17901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:54.408372 17897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:54.408389 17898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:54.408480 17892 server_base.cc:1034] running on GCE node
I20260806 15:58:54.409237 17892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:54.410338 17892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:54.411549 17892 hybrid_clock.cc:648] HybridClock initialized: now 1786031934411521 us; error 38 us; skew 500 ppm
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:54.414448 17892 init.cc:377] Logged in from keytab as kudu/127.13.223.61@KRBTEST.COM (short username kudu)
I20260806 15:58:54.415339 17892 webserver.cc:460] Webserver started at http://127.13.223.61:34325/ using document root <none> and password file <none>
I20260806 15:58:54.415912 17892 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:54.416007 17892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:54.416241 17892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:54.418278 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data/instance:
uuid: "5003bfcfbcb942b1b1c934fc91fd86ae"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "6154352ae4fd068443c9f4b706fb13d6"
server_key_iv: "8da5978e0d34773c10c9b8d5a3af76f5"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.418833 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal/instance:
uuid: "5003bfcfbcb942b1b1c934fc91fd86ae"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "6154352ae4fd068443c9f4b706fb13d6"
server_key_iv: "8da5978e0d34773c10c9b8d5a3af76f5"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.422438 17892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 15:58:54.424978 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:54.426193 17892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 15:58:54.426331 17892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal
uuid: "5003bfcfbcb942b1b1c934fc91fd86ae"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "6154352ae4fd068443c9f4b706fb13d6"
server_key_iv: "8da5978e0d34773c10c9b8d5a3af76f5"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.426436 17892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:54.449035 17892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:54.451968 17892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:54.452200 17892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:54.459287 17892 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.61:45589
I20260806 15:58:54.459585 17999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.61:45589 every 8 connection(s)
I20260806 15:58:54.460263 17892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/data/info.pb
I20260806 15:58:54.461300 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 17892
I20260806 15:58:54.461467 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-1/wal/instance
I20260806 15:58:54.461810 14204 external_mini_cluster.cc:1428] Setting key 4b7e1f00ced72cae69e3de9d2cd139fc
I20260806 15:58:54.463460 18000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:54.468746 18000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.60@KRBTEST.COM" created.
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:54.482666 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.470628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.13.223.62:38441)
0806 15:58:54.470796 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.470801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.471679 (+ 878us) server_negotiation.cc:407] Connection header received
0806 15:58:54.472679 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.472682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.472733 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.472809 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.474166 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.474680 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.476119 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.476367 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.476455 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.479088 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.479091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.479093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.479123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.480613 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.481125 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.481129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.481131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.481211 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.481574 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.481579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.481581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.481761 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.481870 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.482362 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.482491 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 15:58:54.483403 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.470986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38441 (local address 127.0.0.1:38336)
0806 15:58:54.471527 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.471571 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.472434 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.472967 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.472974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.473387 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.474004 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.474028 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.474816 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.474819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.475985 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.475997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.476558 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.476562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.476807 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.477457 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.477473 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.478912 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.480727 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.480733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.480747 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.481013 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.481332 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.481335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.481337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.481476 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.481869 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.481877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.482124 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.482307 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.482628 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":144,"threads_started":1}
I20260806 15:58:54.484419 17884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } attempt: 3
I20260806 15:58:54.484903 18000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab.
Entry for principal kudu/127.13.223.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab.
I20260806 15:58:54.490123 14204 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
/tmp/dist-test-taskBrJzxO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:38571
--webserver_interface=127.13.223.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab
--principal=kudu/127.13.223.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.62:38441,127.13.223.61:45589,127.13.223.60:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
I20260806 15:58:54.496968 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.484882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38806)
0806 15:58:54.485794 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.485809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.485901 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.486241 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.486244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.486431 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.486524 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.486530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.487653 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.487656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.489043 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.489053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.489671 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.489675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.489795 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.490257 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.490273 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.492834 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.495219 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.495223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.495225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.495469 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.495840 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.495843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.495844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.495938 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.496413 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.496416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.496545 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.496653 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.496796 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260806 15:58:54.497066 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.485478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38810)
0806 15:58:54.485537 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.485547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.485613 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.486334 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.486336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.486493 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.486596 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.486604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.487560 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.487565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.488229 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.488238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.489012 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.489018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.489161 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.490024 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.490042 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.492365 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.495433 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.495436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.495439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.495673 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.495958 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.495964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.495969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.496081 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.496591 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.496594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.496692 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.496794 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.496908 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":4}
I20260806 15:58:54.497637 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.485020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.13.223.61:45589)
0806 15:58:54.485340 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.485358 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.485888 (+ 530us) server_negotiation.cc:407] Connection header received
0806 15:58:54.485984 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.485986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.486114 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.486257 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.486639 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.487498 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.489185 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.489508 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.489651 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.493002 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.493005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.493008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.493039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.495106 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.495594 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.495603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.495607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.495709 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.496016 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.496021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.496022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.496301 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.496450 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.497296 (+ 846us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.497504 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":221,"spinlock_wait_cycles":13440,"thread_start_us":142,"threads_started":1}
I20260806 15:58:54.497740 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.485569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.13.223.61:45589)
0806 15:58:54.485846 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.485857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.485883 (+ 26us) server_negotiation.cc:407] Connection header received
0806 15:58:54.485926 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.485934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.486139 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.486303 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.486720 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.487460 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.488340 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.488885 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.488993 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.492577 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.492587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.492599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.492650 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.495318 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.495770 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.495775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.495777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.495858 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.496223 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.496229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.496233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.496475 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.496628 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.496916 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.497459 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":208,"spinlock_wait_cycles":8320,"thread_start_us":139,"threads_started":1}
I20260806 15:58:54.499317 17884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:54.499867 18000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:54.500054 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.499797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34106)
0806 15:58:54.499910 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.499982 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 15:58:54.500797 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.500423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34122)
0806 15:58:54.500621 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.500715 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 15:58:54.500906 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.500630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34128)
0806 15:58:54.500770 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.500816 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260806 15:58:54.501003 17914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
W20260806 15:58:54.501073 17884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:38571: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
I20260806 15:58:54.501957 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.501688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34138)
0806 15:58:54.501815 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.501881 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 15:58:54.502117 18000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:38571: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
I20260806 15:58:54.532235 17884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } attempt: 1
I20260806 15:58:54.533182 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.532825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34150)
0806 15:58:54.532987 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.533057 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 15:58:54.534133 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.533888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34160)
0806 15:58:54.534002 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.534050 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 15:58:54.534346 17884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:38571: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
I20260806 15:58:54.551426 18000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } attempt: 1
I20260806 15:58:54.552388 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.552024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34174)
0806 15:58:54.552213 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.552294 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 15:58:54.553218 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.552962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34190)
0806 15:58:54.553086 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.553152 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 15:58:54.553380 18000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:38571: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
W20260806 15:58:54.609135 18007 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 15:58:54.609431 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 15:58:54.609503 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 15:58:54.609570 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 15:58:54.609580 17884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } attempt: 2
I20260806 15:58:54.610531 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.610202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34194)
0806 15:58:54.610369 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.610441 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 15:58:54.611370 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.611135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34202)
0806 15:58:54.611252 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.611305 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 15:58:54.611586 17884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:38571: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
W20260806 15:58:54.614257 18007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 15:58:54.614315 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 15:58:54.614333 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 15:58:54.614375 18007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 15:58:54.614408 18007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 15:58:54.619861 18007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.62:38441,127.13.223.61:45589,127.13.223.60:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/krb5kdc/kudu_127.13.223.60.keytab
--principal=kudu/127.13.223.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.60:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 15:58:54.621112 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 15:58:54.621577 18000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } attempt: 2
I20260806 15:58:54.622255 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 15:58:54.622440 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.622092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34214)
0806 15:58:54.622281 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.622353 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 15:58:54.623266 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.623012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34216)
0806 15:58:54.623132 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.623196 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 15:58:54.623413 18000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.60:38571: Network error: Client connection negotiation failed: client connection to 127.13.223.60:38571: connect: Connection refused (error 111)
W20260806 15:58:54.631347 18024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:54.631393 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 15:58:54.631433 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 15:58:54.631680 18007 server_base.cc:1034] running on GCE node
I20260806 15:58:54.632287 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 15:58:54.633507 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 15:58:54.634742 18007 hybrid_clock.cc:648] HybridClock initialized: now 1786031934634723 us; error 28 us; skew 500 ppm
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 15:58:54.637738 18007 init.cc:377] Logged in from keytab as kudu/127.13.223.60@KRBTEST.COM (short username kudu)
I20260806 15:58:54.638737 18007 webserver.cc:460] Webserver started at http://127.13.223.60:37251/ using document root <none> and password file <none>
I20260806 15:58:54.639315 18007 fs_manager.cc:359] Metadata directory not provided
I20260806 15:58:54.639411 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 15:58:54.639659 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 15:58:54.641839 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data/instance:
uuid: "3d9ccf6a85e9464299f80016f8a9cb15"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "d806c29157c8a5f2b2b018a5c59b3faa"
server_key_iv: "34df89cac8c94fd39f4bb2bcd43c2416"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.642436 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal/instance:
uuid: "3d9ccf6a85e9464299f80016f8a9cb15"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "d806c29157c8a5f2b2b018a5c59b3faa"
server_key_iv: "34df89cac8c94fd39f4bb2bcd43c2416"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.646427 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 15:58:54.649138 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 15:58:54.650401 18007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 15:58:54.650611 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data,/tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal
uuid: "3d9ccf6a85e9464299f80016f8a9cb15"
format_stamp: "Formatted at 2026-08-06 15:58:54 on dist-test-slave-rwrg"
server_key: "d806c29157c8a5f2b2b018a5c59b3faa"
server_key_iv: "34df89cac8c94fd39f4bb2bcd43c2416"
server_key_version: "encryptionkey@0"
I20260806 15:58:54.650751 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 15:58:54.680111 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 15:58:54.683590 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 15:58:54.683815 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 15:58:54.691495 18007 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.60:38571
I20260806 15:58:54.691812 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.60:38571 every 8 connection(s)
I20260806 15:58:54.692560 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/data/info.pb
I20260806 15:58:54.695358 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 15:58:54.699254 18122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
I20260806 15:58:54.699738 14204 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu as pid 18007
I20260806 15:58:54.699864 14204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBrJzxO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786031632208048-14204-0/minicluster-data/master-2/wal/instance
I20260806 15:58:54.700131 14204 external_mini_cluster.cc:1428] Setting key f22ce8bb7de28fd8989a328fefb11580
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
I20260806 15:58:54.710784 17884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } attempt: 3
I20260806 15:58:54.713943 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.701426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.13.223.62:38441)
0806 15:58:54.701550 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.701553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.702365 (+ 812us) server_negotiation.cc:407] Connection header received
0806 15:58:54.703626 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.703631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.703692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.703801 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.704696 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.705260 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.706250 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.706476 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.706594 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.709457 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.709460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.709462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.709486 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.711260 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.712005 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.712009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.712011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.712107 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.712602 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.712605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.712607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.712840 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.712912 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.713616 (+ 704us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.713753 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 15:58:54.714600 18000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } attempt: 3
I20260806 15:58:54.715412 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.702473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.13.223.60:38571)
0806 15:58:54.702729 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.702740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.702779 (+ 39us) server_negotiation.cc:407] Connection header received
0806 15:58:54.703336 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.703360 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.703638 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.703824 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.704774 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.705727 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.706586 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.706998 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.707126 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.709898 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.709906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.709919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.709950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.712034 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.712533 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.712602 (+ 69us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.712603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.712694 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.712996 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.713000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.713002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.713350 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.713595 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.713806 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.714344 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260806 15:58:54.715409 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.701758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.62:38441 (local address 127.0.0.1:38346)
0806 15:58:54.702206 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.702249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.703336 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.703860 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.703865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.704245 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.704485 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.704553 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.705372 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.705378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.706125 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.706134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.706644 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.706648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.706912 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.707555 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.707579 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.709278 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.711446 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.711453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.711463 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.711874 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.712222 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.712224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.712225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.712426 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.712996 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.713003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.713342 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.714185 (+ 843us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.714396 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":91,"thread_start_us":103,"threads_started":1}
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.62@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
I20260806 15:58:54.717748 18122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
I20260806 15:58:54.723201 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.711248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34228)
0806 15:58:54.711448 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.711464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.711541 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.711917 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.711920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.712100 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.712216 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.712224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.713259 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.713262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.714071 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.714083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.714899 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.714903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.714984 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.715413 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.715429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.717440 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.721548 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.721552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.721557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.721836 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.722154 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.722160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.722165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.722257 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.722708 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.722711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.722803 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.722912 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.723038 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 15:58:54.723243 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.711290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.13.223.60:38571)
0806 15:58:54.711728 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.711734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.711744 (+ 10us) server_negotiation.cc:407] Connection header received
0806 15:58:54.711770 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.711772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.711802 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.711900 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.712349 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.713143 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.714204 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.714502 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.714610 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.718208 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.718212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.718214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.718239 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.720974 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.721941 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.721944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.721946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.722032 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.722370 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.722373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.722376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.722594 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.722714 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.722967 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.723084 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":105,"threads_started":1}
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.61@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.62@KRBTEST.COM
I20260806 15:58:54.727159 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17: Bootstrap starting.
I20260806 15:58:54.727813 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.715077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34230)
0806 15:58:54.715545 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.715560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.715642 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.716444 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.716449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.716671 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.716781 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.716789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.718582 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.718587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.719332 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.719341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.721171 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.721264 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.722002 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.722024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.724148 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.726121 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.726124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.726126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.726394 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.726725 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.726728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.726730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.726817 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.727271 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.727274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.727372 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.727529 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.727656 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384}
I20260806 15:58:54.730113 17884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:54.730731 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.718030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.13.223.62:38441)
0806 15:58:54.718133 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.718137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.718365 (+ 228us) server_negotiation.cc:407] Connection header received
0806 15:58:54.718404 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.718410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.718446 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.718573 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.719521 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.720165 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721031 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.721331 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.727290 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.727297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.727300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.727324 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.728918 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.729408 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.729413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.729415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.729498 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.729867 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.729871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.729872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.730046 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.730321 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.730401 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.730510 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 15:58:54.731048 17884 log.cc:824] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:54.731812 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.715544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.13.223.60:38571)
0806 15:58:54.716198 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.716220 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.716233 (+ 13us) server_negotiation.cc:407] Connection header received
0806 15:58:54.716267 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.716276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.716309 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.716379 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.716918 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.718068 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.719817 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.720184 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721138 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.724278 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.724281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.724284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.724307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.725968 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.726521 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.726524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.726526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.726612 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.726933 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.726935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.726937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.727148 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.728252 (+ 1104us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.731470 (+ 3218us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.731594 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"spinlock_wait_cycles":1851648,"thread_start_us":156,"threads_started":1}
I20260806 15:58:54.732900 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.718156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.61:45589 (local address 127.0.0.1:38816)
0806 15:58:54.718256 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.718280 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.718343 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.718745 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.718751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.718946 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.719086 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.719094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.720107 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.720111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.720985 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.721004 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.721580 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.721651 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.722010 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.722034 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.724147 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.726411 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.726417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.726419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.726655 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.727006 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.727022 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.727024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.727122 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.728186 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.728193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.729116 (+ 923us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.732401 (+ 3285us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x3d8eacb8. stack: 00007fdaf5aa4c78 00007fdaf5aa51a9 00007fdaf56671d7 00000000005984a6 00000000005984f9 000000000059c488 00007fdaf5aa12ab 00007fdaf5aa09da 00007fdafa1790c4 00007fdafa17969a 00007fdafa0fcf15 00007fdafa0f7ff0 00007fdafa138832 00007fdafa13968c 00007fdafa14f0d3 00007fdafa15254b
0806 15:58:54.732598 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.732738 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"spinlock_wait_cycles":5286400}
I20260806 15:58:54.732923 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.718398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.13.223.61:45589)
0806 15:58:54.718558 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.718564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.718576 (+ 12us) server_negotiation.cc:407] Connection header received
0806 15:58:54.718605 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.718607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.718646 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.718720 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.719217 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.719984 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721125 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.721399 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.721457 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.724329 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.724335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.724339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.724368 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.726277 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.726781 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.726784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.726786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.726873 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.727243 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.727248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.727250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.728093 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.728186 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.732659 (+ 4473us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.732784 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 15:58:54.733340 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17: No bootstrap required, opened a new log
I20260806 15:58:54.737298 18122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.61@KRBTEST.COM
I20260806 15:58:54.737617 17884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.737848 17884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770c099adf844e7896891ac13e4a6e17, State: Initialized, Role: FOLLOWER
I20260806 15:58:54.738207 17884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.739499 17884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:54.739835 18000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: Bootstrap starting.
I20260806 15:58:54.739060 18133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.740012 18133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.741818 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.732250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.13.223.61:45589)
0806 15:58:54.732361 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.732365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.732379 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:54.732413 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.732416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.732455 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.732576 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.733434 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.733949 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.734973 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.735207 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.735288 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.738020 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.738023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.738026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.738051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.739693 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.740176 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.740179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.740181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.740267 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.740636 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.740640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.740642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.740891 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.741086 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.741476 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.741595 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 15:58:54.743379 18000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:54.744283 18000 log.cc:824] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: Log is configured to *not* fsync() on all Append() calls
Aug 06 15:58:54 dist-test-slave-rwrg krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786031934, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.60@KRBTEST.COM for kudu/127.13.223.60@KRBTEST.COM
I20260806 15:58:54.747119 18000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: No bootstrap required, opened a new log
I20260806 15:58:54.748973 14204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 15:58:54.745036 17776 master_runner.cc:186] Error getting master registration for 127.13.223.61:45589: OK, instance_id { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" instance_seqno: 1786031934457391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 15:58:54.749243 17776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 15:58:54.750248 17839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "770c099adf844e7896891ac13e4a6e17"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38350
I20260806 15:58:54.750473 17839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 15:58:54.750519 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.740784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.60:38571 (local address 127.0.0.1:34232)
0806 15:58:54.741000 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 15:58:54.741014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 15:58:54.741085 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 15:58:54.741783 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 15:58:54.741785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 15:58:54.741953 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 15:58:54.742047 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.742054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.743336 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.743340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.744144 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 15:58:54.744156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.744875 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.744878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 15:58:54.744958 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.745419 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 15:58:54.745438 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 15:58:54.746939 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 15:58:54.748965 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.748968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.748970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.749190 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.749511 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 15:58:54.749515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 15:58:54.749517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 15:58:54.749633 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 15:58:54.750068 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 15:58:54.750072 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 15:58:54.750148 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 15:58:54.750226 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 15:58:54.750353 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 15:58:54.750635 17839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:54.751155 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.741217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.13.223.60:38571)
0806 15:58:54.741533 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.741539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.741553 (+ 14us) server_negotiation.cc:407] Connection header received
0806 15:58:54.741584 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.741587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.741639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.742370 (+ 731us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.742460 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.743206 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.744344 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.744751 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.745172 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.747082 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.747085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.747088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.747112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.748850 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.749315 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.749318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.749320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.749393 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.749743 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.749747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.749748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.749958 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.750834 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.750904 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.751016 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260806 15:58:54.751566 18000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.751811 18000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5003bfcfbcb942b1b1c934fc91fd86ae, State: Initialized, Role: FOLLOWER
I20260806 15:58:54.752003 17839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.752144 18000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.752982 18157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.753031 17839 leader_election.cc:288] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [CANDIDATE]: Term 1 election: Requested vote from peers 5003bfcfbcb942b1b1c934fc91fd86ae (127.13.223.61:45589), 3d9ccf6a85e9464299f80016f8a9cb15 (127.13.223.60:38571)
I20260806 15:58:54.753175 18157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.753839 18000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:54.753918 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "770c099adf844e7896891ac13e4a6e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae"
I20260806 15:58:54.754005 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: Bootstrap starting.
I20260806 15:58:54.754148 17947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:54.755707 17947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770c099adf844e7896891ac13e4a6e17 in term 1.
I20260806 15:58:54.756110 17800 leader_election.cc:302] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5003bfcfbcb942b1b1c934fc91fd86ae, 770c099adf844e7896891ac13e4a6e17; no voters:
I20260806 15:58:54.756397 18133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 15:58:54.756641 18133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [term 1 LEADER]: Becoming Leader. State: Replica: 770c099adf844e7896891ac13e4a6e17, State: Running, Role: LEADER
I20260806 15:58:54.756729 18122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: Neither blocks nor log segments found. Creating new log.
I20260806 15:58:54.756942 18133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
W20260806 15:58:54.757534 18155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:54.757591 18155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:54.757642 18122 log.cc:824] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: Log is configured to *not* fsync() on all Append() calls
I20260806 15:58:54.759128 18163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770c099adf844e7896891ac13e4a6e17. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.759243 18163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: This master's current role is: LEADER
W20260806 15:58:54.761765 18174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:54.761853 18174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:54.761981 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: No bootstrap required, opened a new log
I20260806 15:58:54.765726 18190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 15:58:54.769385 18190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 15:58:54.766687 17892 master_runner.cc:186] Error getting master registration for 127.13.223.60:38571: OK, instance_id { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" instance_seqno: 1786031934689393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 15:58:54.770694 17892 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 15:58:54.772172 18122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.772456 18122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9ccf6a85e9464299f80016f8a9cb15, State: Initialized, Role: FOLLOWER
I20260806 15:58:54.772902 18122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } }
I20260806 15:58:54.773113 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.759948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.13.223.61:45589)
0806 15:58:54.760051 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.760056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.763551 (+ 3495us) server_negotiation.cc:407] Connection header received
0806 15:58:54.763672 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.763675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.763725 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.763803 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.764836 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.765533 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.766718 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.767056 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.767132 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.769026 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.769030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.769047 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.769075 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.770879 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.771486 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.771489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.771491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.771582 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.771892 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.771895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.771897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.772120 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.772254 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.772826 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.772948 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 15:58:54.774380 18122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 15:58:54.774744 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "770c099adf844e7896891ac13e4a6e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d9ccf6a85e9464299f80016f8a9cb15"
I20260806 15:58:54.775007 18082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 15:58:54.776439 18082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770c099adf844e7896891ac13e4a6e17 in term 1.
I20260806 15:58:54.776682 18199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.776822 18199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.787559 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.777854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.13.223.60:38571)
0806 15:58:54.778001 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.778007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.778474 (+ 467us) server_negotiation.cc:407] Connection header received
0806 15:58:54.778516 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.778524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.778589 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.778678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.779467 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.780263 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.781348 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.781703 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.781792 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.783569 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.783576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.783580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.783609 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.785393 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.785874 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.785880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.785883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.785972 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.786294 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.786298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.786299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.786533 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.786682 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.787285 (+ 603us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.787405 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 15:58:54.793812 18221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 15:58:54.793900 18221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 15:58:54.794138 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:54.778148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.13.223.62:38441)
0806 15:58:54.778288 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.778293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.779046 (+ 753us) server_negotiation.cc:407] Connection header received
0806 15:58:54.779232 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.779236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.779283 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.779338 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.780236 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.781047 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.788794 (+ 7747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:54.789076 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:54.789139 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:54.790613 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 15:58:54.790617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 15:58:54.790620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 15:58:54.790646 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 15:58:54.792154 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.792699 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.792704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.792708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.792795 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 15:58:54.793165 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 15:58:54.793169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 15:58:54.793171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 15:58:54.793387 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 15:58:54.793487 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:54.793846 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 15:58:54.793973 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 15:58:54.795037 18082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [term 1 FOLLOWER]: Refusing update from remote peer 770c099adf844e7896891ac13e4a6e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:54.795774 18163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:54.796360 17947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [term 1 FOLLOWER]: Refusing update from remote peer 770c099adf844e7896891ac13e4a6e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 15:58:54.796933 18163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 15:58:54.799664 18199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770c099adf844e7896891ac13e4a6e17. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.799787 18199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.801389 18157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770c099adf844e7896891ac13e4a6e17. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.801523 18157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.807274 18133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.807394 18133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:54.807798 18157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.807933 18157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.809222 18133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.809356 18133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17 [sys.catalog]: This master's current role is: LEADER
I20260806 15:58:54.810228 18199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "770c099adf844e7896891ac13e4a6e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770c099adf844e7896891ac13e4a6e17" member_type: VOTER last_known_addr { host: "127.13.223.62" port: 38441 } } peers { permanent_uuid: "5003bfcfbcb942b1b1c934fc91fd86ae" member_type: VOTER last_known_addr { host: "127.13.223.61" port: 45589 } } peers { permanent_uuid: "3d9ccf6a85e9464299f80016f8a9cb15" member_type: VOTER last_known_addr { host: "127.13.223.60" port: 38571 } } }
I20260806 15:58:54.810343 18199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 15:58:54.811152 18224 mvcc.cc:205] Tried to move back new op lower bound from 7315586804901040128 to 7315586804767772672. Current Snapshot: MvccSnapshot[applied={T|T < 7315586804901040128}]
I20260806 15:58:54.816354 18190 catalog_manager.cc:1347] Generated new cluster ID: d9f52a7d28404f139f3c87eded3eb41e
I20260806 15:58:54.816426 18190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 15:58:54.822311 18228 mvcc.cc:205] Tried to move back new op lower bound from 7315586804901040128 to 7315586804767772672. Current Snapshot: MvccSnapshot[applied={T|T < 7315586804901040128}]
I20260806 15:58:54.849982 18190 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 15:58:54.851425 18190 catalog_manager.cc:1504] Loading token signing keys...
I20260806 15:58:54.871774 18190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770c099adf844e7896891ac13e4a6e17: Generated new TSK 0
I20260806 15:58:54.872720 18190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 15:58:54.907730 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.904517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.13.223.61:45589)
0806 15:58:54.904662 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.904667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.906615 (+ 1948us) server_negotiation.cc:407] Connection header received
0806 15:58:54.906838 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.906842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.906892 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.906946 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.907581 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38848: BlockingRecv error: recv got EOF from 127.0.0.1:38848 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 15:58:54.909116 18130 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.905048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.13.223.60:38571)
0806 15:58:54.905163 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.905167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.908206 (+ 3039us) server_negotiation.cc:407] Connection header received
0806 15:58:54.908370 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.908373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.908421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.908470 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.908989 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34250: BlockingRecv error: recv got EOF from 127.0.0.1:34250 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 15:58:54.910454 17888 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.905482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.13.223.62:38441)
0806 15:58:54.905595 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.905600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.909600 (+ 4000us) server_negotiation.cc:407] Connection header received
0806 15:58:54.909785 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.909788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.909837 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.909885 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.910338 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38358: BlockingRecv error: recv got EOF from 127.0.0.1:38358 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 15:58:54.915012 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.913395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.13.223.61:45589)
0806 15:58:54.913499 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.913503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.914034 (+ 531us) server_negotiation.cc:407] Connection header received
0806 15:58:54.914261 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.914265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.914340 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.914393 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.914912 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38850: BlockingRecv error: recv got EOF from 127.0.0.1:38850 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 15:58:54.919960 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.918601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.13.223.61:45589)
0806 15:58:54.918710 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.918714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.919071 (+ 357us) server_negotiation.cc:407] Connection header received
0806 15:58:54.919249 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.919252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.919298 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.919345 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.919864 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38864: BlockingRecv error: recv got EOF from 127.0.0.1:38864 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 15:58:54.925956 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.924164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.13.223.61:45589)
0806 15:58:54.924275 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.924279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.925024 (+ 745us) server_negotiation.cc:407] Connection header received
0806 15:58:54.925229 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.925233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.925290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.925344 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.925863 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38868: BlockingRecv error: recv got EOF from 127.0.0.1:38868 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 15:58:54.934069 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.932274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.13.223.61:45589)
0806 15:58:54.932416 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.932421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.933113 (+ 692us) server_negotiation.cc:407] Connection header received
0806 15:58:54.933320 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.933324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.933378 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.933429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.933957 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38882: BlockingRecv error: recv got EOF from 127.0.0.1:38882 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 15:58:54.946548 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.944710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.13.223.61:45589)
0806 15:58:54.944852 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.944857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.945467 (+ 610us) server_negotiation.cc:407] Connection header received
0806 15:58:54.945714 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.945718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.945777 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.945832 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.946434 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38898: BlockingRecv error: recv got EOF from 127.0.0.1:38898 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 15:58:54.966962 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:54.965234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.13.223.61:45589)
0806 15:58:54.965363 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:54.965368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:54.965952 (+ 584us) server_negotiation.cc:407] Connection header received
0806 15:58:54.966159 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:54.966165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:54.966222 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:54.966274 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:54.966838 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38908: BlockingRecv error: recv got EOF from 127.0.0.1:38908 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 15:58:55.003073 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:55.001731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.13.223.61:45589)
0806 15:58:55.001872 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.001879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.002113 (+ 234us) server_negotiation.cc:407] Connection header received
0806 15:58:55.002305 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.002308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.002367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.002459 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:55.002937 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38910: BlockingRecv error: recv got EOF from 127.0.0.1:38910 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 15:58:55.070771 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:55.069426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.13.223.61:45589)
0806 15:58:55.069634 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.069639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.069770 (+ 131us) server_negotiation.cc:407] Connection header received
0806 15:58:55.069962 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.069965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.070014 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.070109 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:55.070626 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38912: BlockingRecv error: recv got EOF from 127.0.0.1:38912 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 15:58:55.202457 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:55.201139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.13.223.61:45589)
0806 15:58:55.201278 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.201283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.201465 (+ 182us) server_negotiation.cc:407] Connection header received
0806 15:58:55.201662 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.201668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.201736 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.201835 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:55.202312 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38916: BlockingRecv error: recv got EOF from 127.0.0.1:38916 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 15:58:55.461915 18005 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 15:58:55.460770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.13.223.61:45589)
0806 15:58:55.460910 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.460915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.461037 (+ 122us) server_negotiation.cc:407] Connection header received
0806 15:58:55.461178 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.461181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.461238 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.461337 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 15:58:55.461808 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38918: BlockingRecv error: recv got EOF from 127.0.0.1:38918 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 15:58:55.763161 18174 catalog_manager.cc:1259] Loaded cluster ID: d9f52a7d28404f139f3c87eded3eb41e
I20260806 15:58:55.763250 18174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: loading cluster ID for follower catalog manager: success
I20260806 15:58:55.765021 18174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: acquiring CA information for follower catalog manager: success
I20260806 15:58:55.766222 18174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5003bfcfbcb942b1b1c934fc91fd86ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 15:58:55.795266 18221 catalog_manager.cc:1259] Loaded cluster ID: d9f52a7d28404f139f3c87eded3eb41e
I20260806 15:58:55.795361 18221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: loading cluster ID for follower catalog manager: success
I20260806 15:58:55.797205 18221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: acquiring CA information for follower catalog manager: success
I20260806 15:58:55.798426 18221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d9ccf6a85e9464299f80016f8a9cb15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 15:58:55.980412 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:55.976351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.13.223.61:45589)
0806 15:58:55.976667 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.976673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.976692 (+ 19us) server_negotiation.cc:407] Connection header received
0806 15:58:55.976871 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.976875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.977015 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.977141 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:55.977443 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.978005 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.978885 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.979131 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.979239 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:55.979560 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:55.979998 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:55.980135 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:55.980199 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 15:58:55.980255 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260806 15:58:55.987408 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:55.983696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.13.223.60:38571)
0806 15:58:55.983949 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.983961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.984071 (+ 110us) server_negotiation.cc:407] Connection header received
0806 15:58:55.984226 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.984228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.984339 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.984429 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:55.984759 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.985230 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.986062 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.986258 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.986327 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:55.986573 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:55.986950 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:55.987037 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:55.987178 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 15:58:55.987222 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260806 15:58:55.994414 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:55.990312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.13.223.61:45589)
0806 15:58:55.990428 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.990432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.990828 (+ 396us) server_negotiation.cc:407] Connection header received
0806 15:58:55.991012 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.991016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.991069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.991144 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:55.991377 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.991983 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.993148 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.993489 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.993599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:55.993923 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:55.994002 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:55.994088 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:55.994213 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 15:58:55.994265 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 15:58:55.994491 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:55.990382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.13.223.60:38571)
0806 15:58:55.990519 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.990523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.990894 (+ 371us) server_negotiation.cc:407] Connection header received
0806 15:58:55.991065 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.991068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.991135 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.991198 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:55.991566 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.992156 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.993189 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.993522 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.993620 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:55.994042 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:55.994114 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:55.994163 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:55.994317 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 15:58:55.994366 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 15:58:55.995714 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 15:58:55.990233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.13.223.62:38441)
0806 15:58:55.990469 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 15:58:55.990475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 15:58:55.990998 (+ 523us) server_negotiation.cc:407] Connection header received
0806 15:58:55.991279 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 15:58:55.991283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 15:58:55.991441 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 15:58:55.991555 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 15:58:55.991925 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.992699 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.993733 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 15:58:55.994096 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 15:58:55.994214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 15:58:55.994961 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 15:58:55.995294 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 15:58:55.995390 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 15:58:55.995460 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 15:58:55.995511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
I20260806 15:58:56.004019 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17776
I20260806 15:58:56.015951 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 17892
I20260806 15:58:56.031455 14204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBrJzxO/build/debug/bin/kudu with pid 18007
2026-08-06T15:58:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3349 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6708 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (303793 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST